Massive Data Size Research Articles

Scalable Bayesian uncertainty quantification with data-driven priors for radio interferometric imaging

Abstract Next-generation radio interferometers like the Square Kilometer Array have the potential to unlock scientific discoveries thanks to their unprecedented angular resolution and sensitivity. One key to unlocking their potential resides in handling the deluge and complexity of incoming data. This challenge requires building radio interferometric (RI) imaging methods that can cope with the massive data sizes and provide high-quality image reconstructions with uncertainty quantification (UQ). This work proposes a method coined QuantifAI to address UQ in RI imaging with data-driven (learned) priors for high-dimensional settings. Our model, rooted in the Bayesian framework, uses a physically motivated model for the likelihood. The model exploits a data-driven convex prior potential, which can encode complex information learned implicitly from simulations and guarantee the log-concavity of the posterior. We leverage probability concentration phenomena of high-dimensional log-concave posteriors to obtain information about the posterior, avoiding MCMC sampling techniques. We rely on convex optimisation methods to compute the MAP estimation, which is known to be faster and better scale with dimension than MCMC strategies. QuantifAI allows us to compute local credible intervals and perform hypothesis testing of structure on the reconstructed image. We propose a novel fast method to compute pixel-wise uncertainties at different scales, which uses 3 and 6 orders of magnitude less likelihood evaluations than other UQ methods like length of the credible intervals and Monte Carlo posterior sampling, respectively. We demonstrate our method by reconstructing RI images in a simulated setting and carrying out fast and scalable UQ, which we validate with MCMC sampling. Our method shows an improved image quality and more meaningful uncertainties than the benchmark method based on a sparsity-promoting prior. QuantifAI’s source code is available from https://github.com/astro-informatics/QuantifAI.

Comprehensive framework of GPU-accelerated image reconstruction for photoacoustic computed tomography.

Photoacoustic computed tomography (PACT) is a promising non-invasive imaging technique for both life science and clinical implementations. To achieve fast imaging speed, modern PACT systems have equipped arrays that have hundreds to thousands of ultrasound transducer (UST) elements, and the element number continues to increase. However, large number of UST elements with parallel data acquisition could generate a massive data size, making it very challenging to realize fast image reconstruction. Although several research groups have developed GPU-accelerated method for PACT, there lacks an explicit and feasible step-by-step description of GPU-based algorithms for various hardware platforms. In this study, we propose a comprehensive framework for developing GPU-accelerated PACT image reconstruction (GPU-accelerated photoacoustic computed tomography), to help the research community to grasp this advanced image reconstruction method. We leverage widely accessible open-source parallel computing tools, including Python multiprocessing-based parallelism, Taichi Lang for Python, CUDA, and possible other backends. We demonstrate that our framework promotes significant performance of PACT reconstruction, enabling faster analysis and real-time applications. Besides, we also described how to realize parallel computing on various hardware configurations, including multicore CPU, single GPU, and multiple GPUs platform. Notably, our framework can achieve an effective rate of times when reconstructing extremely large-scale three-dimensional PACT images on a dual-GPU platform compared to a 24-core workstation CPU. In this paper, we share example codes via GitHub. Our approach allows for easy adoption and adaptation by the research community, fostering implementations of PACT for both life science and medicine.

A Survey on Non-photorealistic Rendering Approaches for Point Cloud Visualization.

Point clouds are widely used as a versatile representation of 3D entities and scenes for all scale domains and in a variety of application areas, serving as a fundamental data category to directly convey spatial features. However, due to point sparsity, lack of structure, irregular distribution, and acquisition-related inaccuracies, results of point cloud visualization are often subject to visual complexity and ambiguity. In this regard, non-photorealistic rendering can improve visual communication by reducing the cognitive effort required to understand an image or scene and by directing attention to important features. In the last 20 years, this has been demonstrated by various non-photorealistic rendering approaches that were proposed to target point clouds specifically. However, they do not use a common language or structure for assessment which complicates comparison and selection. Further, recent developments regarding point cloud characteristics and processing, such as massive data size or web-based rendering are rarely considered. To address these issues, we present a survey on non-photorealistic rendering approaches for point cloud visualization, providing an overview of the current state of research. We derive a structure for the assessment of approaches, proposing seven primary dimensions for the categorization regarding intended goals, data requirements, used techniques, and mode of operation. We then systematically assess corresponding approaches and utilize this classification to identify trends and research gaps, motivating future research in the development of effective non-photorealistic point cloud rendering methods.

Super-resolution network with dynamic cleanup and temporal–spatial attention for compressed videos

Video super-resolution methods focus on restoring high-resolution video frames from low-resolution videos with pre-defined degradations, and few consider compression. However, the videos on the Internet are compressed to reduce the massive size of video data. Thus, they also contain visual noises and artifacts due to the lossy compression. In this paper, we propose a video super-resolution model specific for compressed videos to suppress artifacts and restore high-resolution contents. We propose a two-stage super-resolution network for compressed videos that contains three new modules: cleanup and local alignment in the first stage, 3D-convolution temporal–spatial attention in the second stage. Specifically, the cleanup module is designed in stage-1 to suppress the artifacts and noises of low-quality inputs. The subsequent local alignment is designed to aggregate adjacent frames to maintain high-frequency details. The 3D-convolution attention is used in stage-2 to refine the weights of the feature maps. In addition, we propose the new CrfVideos dataset to facilitate fair comparisons, containing HEVC/H.265 compressed videos of different compression levels. Extensive experiments substantiate that our method outperforms state-of-the-art methods on the HEVC/H.265 compressed benchmark with medium to high compression rates by 0.3–0.7 dB in terms of PSNR and exceeds state-of-the-art methods on benchmark Vid4 by 0.2–0.8 dB. The code will be released at https://github.com/cvygkhv/decompression.

Trident: A Deep Learning Framework for High-Resolution Bridge Vibration Monitoring

Bridges are the essential components in lifeline transportation systems, and their safe operation is of great importance. Information on structural damage could assist in timely repairs and reduce downtime. With the latest advancements in sensing technology, collecting vibration data from bridges has become more accessible. However, effective vibration processing is still a challenge, given the high dimensionality and massive size of vibration data. Existing studies have shown that machine/deep learning techniques can be valuable tools for this task. However, the learning and computational capacities of these models are challenged in the presence of large sensor arrays. We propose Trident as a novel deep learning framework that enables automatic damage feature extraction by simultaneously learning from temporal and three-dimensional (3D) spatial variations of 6D input data in instrumented bridges. Trident is equipped with 3 ConvLSTM3D branches to achieve this goal. A 3D steel truss bridge subject to dynamic traffic loads is monitored for its vibrations to evaluate Trident’s robustness in finding damaged elements. A damage dataset of 52,800 vehicle passing simulations is generated leveraging a database of 528 passenger vehicles in the United States, obtained from the National Highway and Traffic Safety Administration. Bayesian optimization is utilized to tune the model’s hyperparameters, achieving a test Node Average Geometric Mean Accuracy of 86%. This level of performance is promising given the high dimensionality and complexities of the output space in vibration-based monitoring. Trident’s concept can be extended to other vibration monitoring tasks with different time series data and damage labeling strategies.

Spatial-Temporal Analysis of Multi-Subject Functional Magnetic Resonance Imaging Data

Functional magnetic resonance imaging (fMRI) is one of the most popular neuroimaging technologies used in human brain studies. However, fMRI data analysis faces several challenges, including intensive computation due to the massive data size and large estimation errors due to a low signal-to-noise ratio of the data. A new statistical model and a computational algorithm are proposed to address these challenges. Specifically, a new multi-subject general linear model is built for stimulus-evoked fMRI data. The new model assumes that brain responses to stimuli at different brain regions of various subjects fall into a low-rank structure and can be represented by a few principal functions. Therefore, the new model enables combining data information across subjects and regions to evaluate subject-specific and region-specific brain activity. Two optimization functions and a new fast-to-compute algorithm are developed to analyze multi-subject stimulus-evoked fMRI data and address two research questions of a broad interest in psychology: evaluating every subject’s brain responses to different stimuli and identifying brain regions responsive to the stimuli. Both simulation and real data analysis are conducted to show that the new method can outperform existing methods by providing more efficient estimates of brain activity.

Fast Error-Bounded Distance Distribution Computation

In this work we study the distance distribution computation problem. It has been widely used in many real-world applications, e.g., human genome clustering, cosmological model analysis, and parameter tuning. The straightforward solution for the exact distance distribution computation problem is unacceptably slow due to (i) massive data size, and (ii) expensive distance computation. In this paper, we propose a novel method to compute approximate distance distributions with error bound guarantees. Furthermore, our method is generic to different distance measures. We conduct extensive experimental studies on three widely used distance measures with real-world datasets. The experimental results demonstrate that our proposed method outperforms the sampling-based solution (without error guarantees) by up to three orders of magnitude.

Self-Organized Model Fitting Method for Railway Structures Monitoring Using LiDAR Point Cloud

Mobile laser scanning (MLS) has been successfully used for infrastructure monitoring apt to its fine accuracy and higher point density, which is favorable for object reconstruction. The massive data size, computational time, wider spatial distribution and feature extraction become a challenging task for 3D point data processing with MLS point cloud receives from terrestrial structures such as buildings, roads and railway tracks. In this paper, we propose a new approach to detect the structures in-line with railway track geometry such as railway crossings, turnouts and quantitatively estimate their dimensions and spatial location by iteratively applying a vertical slice to point cloud data for long distance laser measurement. The rectangular vertical slices were defined and their boundary coordinates were estimated based on a geometrical method. Estimated vertical slice boundaries were iteratively used to evaluate the point density of each vertical slice along with a cross-track direction of the railway line. Those point densities were further analyzed to detect the railway line track objects by their shape and spatial location along with the rail bed. Herein, the survey dataset is used as a dictionary to preidentify the spatial location of the object and then as an accurate estimation for the rail-track, by estimating the gauge corner (GC) from dense point cloud. The proposed method has shown a significant improvement in the rail-track extraction process, which becomes a challenge for existing remote sensing technologies. This adaptive object detection method can be used to identify the railway track structures prior to the railway track extraction, which allows in finding the GC position precisely. Further, it is based on the parallelism of the railway track, which is distinct from conventional railway track extraction methods. Therefore it does not require any inertial measurements along with the MLS survey and can be applied with less background information of the observed MLS point cloud. The proposed algorithm was tested for the MLS data set acquired during the pilot project collaborated with West Japan Railway Company. The results indicate 100% accuracy for railway structure detection and enhance the GC extraction for railway structure monitoring.

Hardware Performance Evaluation of De novo Transcriptome Assembly Software in Amazon Elastic Compute Cloud

Background:Bioinformatics software for RNA-seq analysis has a high computational requirement in terms of the number of CPUs, RAM size, and processor characteristics. Specifically, de novo transcriptome assembly demands large computational infrastructure due to the massive data size, and complexity of the algorithms employed. Comparative studies on the quality of the transcriptome yielded by de novo assemblers have been previously published, lacking, however, a hardware efficiency-oriented approach to help select the assembly hardware platform in a cost-efficient way.Objective:We tested the performance of two popular de novo transcriptome assemblers, Trinity and SOAPdenovo-Trans (SDNT), in terms of cost-efficiency and quality to assess limitations, and provided troubleshooting and guidelines to run transcriptome assemblies efficiently.Methods:We built virtual machines with different hardware characteristics (CPU number, RAM size) in the Amazon Elastic Compute Cloud of the Amazon Web Services. Using simulated and real data sets, we measured the elapsed time, cost, CPU percentage and output size of small and large data set assemblies.Results:For small data sets, SDNT outperformed Trinity by an order the magnitude, significantly reducing the time duration and costs of the assembly. For large data sets, Trinity performed better than SDNT. Both the assemblers provide good quality transcriptomes.Conclusion:The selection of the optimal transcriptome assembler and provision of computational resources depend on the combined effect of size and complexity of RNA-seq experiments.

An automated approach for street trees detection using mobile laser scanner data

The trees on the street are the primary element of urban vegetation and essential for environmental conservation. Environmental quality improvement and offering amenities for inhabitants are the fundamental roles of street trees. The visual magnificence of urban landscape is also reliant on them. Accurate, conversant inventory information and monitoring conditions are necessary for urban horticultural planning, maintenance, for land use and land cover management. In this propound research work, an automated method for detecting the street trees utilizing Mobile Laser Scanner (MLS) data has been proposed. The proposed method consists of four phases. The first phase divides the MLS dataset into regular two-dimensional square grids. Only eligible grids are further sliced into vertical layers in the second phase. Region growing is been performed to detect the street trees in the third phase of methodology. Elimination of redundant tree points has been performed in the last phase of proposed methodology. This approach has been tested on a MLS dataset. The result shows the precision, recall and F Score of the method for street trees detection as 97.64%, 96.77% and 0.9729 respectively. Significance of Present Study Large number of trees are evacuated during road broadening and maintenance. Therefore, creation of up-to-date street trees inventory is required for proper monitoring and planning of their growth rate, place and proximity in order to save them. Mobile laser scanning facilitates the mapping of road side objects at high speed. However, till now lots of manual work has been presented in processing of Lidar dataset and the bottleneck processing has shifted from the data collection phase to data processing phase. Often number of days are disbursed on processing of data which are captured in couple of hours due to the massive data size. In such scenario to analyze a particular kind of geo-spatial feature preferably would be better to extract feature from the huge data size. Therefore, automated and time efficient extraction of geospatial feature is very indispensable. In the present study an automated method for the detection of street trees has been proposed. Manual processing of Lidar dataset for the identification of trees has been nearly eliminated along with less time complexity and good accuracy. The major implication of this research is that it can automatically, time efficiently and accurately detect the street trees from entire big and bulky captured Lidar point cloud dataset. The urban environmental conservation, monitoring, development and other related authorities shall be considerably benefitted from this research.

Portable interactive visualization of large-scale simulations in geotechnical engineering using Unity3D

Development in large-scale geotechnical engineering simulation places tremendous demand for efficient visualization of such simulation data. This study presents a lightweight software tool, i.e. Geotechnical Interactive Visualization (GIV), as a solution to this challenge, which achieves efficient interactive visualization of large-scale simulations data in geotechnical engineering. Visualization data flow and algorithms specifically optimized for common geotechnical engineering applications are implemented in GIV. GIV can visualize geotechnical structure models with time-varying attributes attached to mesh with fixed topology, and also models with time-varying mesh topologies but no attributes attached, the two most common visualization tasks in geotechnical engineering. Challenges for large-scale simulation data visualization, including parallel simulation data redundancy, massive data size, dynamic user interaction, and portability are overcome via specifically designed algorithms for simulation data preprocessing and optimized visualization modules using the powerful 3D rendering and interactive game engine Unity3D. Comparison of GIV with several widely used visualization tools for the visualization of large-scale idealized datasets and realistic geotechnical simulations highlights the visualization efficiency, smooth interactivity, and lightweight features of GIV.

Fog Computing Based Hybrid Deep Learning Framework in effective inspection system for smart manufacturing

Most sensors have been taken up, which resulted in a massive data size, with the continuously growing IoT (Internet of Things) devices and communications infrastructure in development. The inspection of the manufacturer to identify product defects is one of the most common examples. In order to develop an effective inspection system with greater precision, this paper has been proposed a Fog Computing based Hybrid Deep-Learning Framework (FC-HDLF), that can find possible defective products. Since a large number of assembly lines can occur in a single factory, one of the main problems is how these data are processed in real-time. The system can handle incredibly large amounts of data by discharging the load from the central servers to the fog nodes. In this paper, there are two obvious advantages. Next, the Convolutional Neural Network (CNN) model is adapted to the fog computing environment, which improves its calculation performance considerably. The other is that a model of control is built that can display the form and extent of the defect simultaneously. A decision-making framework for multi-agents is built to ensure a production process architecture to optimize production processes.

A Combined Horizontal Parallel Apriori Algorithm and Adaptive Frequent Pattern Growth Algorithm for Big Data Mining

Due to the massive data size and complexness, big data mining using a sole computer is a problematic task. With the rapid increase in the database size, parallel and distributed computing systems can yield better benefits in the data mining applications. Parallelization of the Association Rule Mining (ARM) algorithms is a significant task in the data mining application for effectively mining the frequent itemsets from the large-size databases. These mining algorithms allocate the database in a horizontal manner or increase the number of processors to decrease the overall time necessary for mining the frequent itemsets. In this paper, a combined Horizontal Parallel-Apriori (HP-Apriori) and Adaptive Frequent Pattern (FP) Growth algorithm is proposed to divide the database both horizontally and vertically into four sub-processes, for parallel processing of all four tasks. The Horizontal Parallel-Apriori algorithm increases the speed of the mining process using an index file. Adaptive Binomial Distribution (ABD) is applied to the Frequent Pattern Growth Algorithm to find the minimum support for mining the optimal frequent itemsets. Experimental analysis established that the combined algorithm outperforms in terms of minimizing the overall execution time and increasing the computational speed in high scalability.

A Review of Content-Based Video Retrieval Techniques for Person Identification

The rise of technology spurs the advancement in the surveillance field. Many commercial spaces reduced the patrol guard in favor of Closed-Circuit Television (CCTV) installation and even some countries already used surveillance drone which has greater mobility. In recent years, the CCTV Footage have also been used for crime investigation by law enforcement such as in Boston Bombing 2013 incident. However, this led us into producing huge unmanageable footage collection, the common issue of Big Data era. While there is more information to identify a potential suspect, the massive size of data needed to go over manually is a very laborious task. Therefore, some researchers proposed using Content-Based Video Retrieval (CBVR) method to enable to query a specific feature of an object or a human. Due to the limitations like visibility and quality of video footage, only certain features are selected for recognition based on Chicago Police Department guidelines. This paper presents the comprehensive reviews on CBVR techniques used for clothing, gender and ethnic recognition of the person of interest and how can it be applied in crime investigation. From the findings, the three recognition types can be combined to create a Content-Based Video Retrieval system for person identification.

PowerHash: a hybrid grouping scheme by leveraging power-law properties of data

We study GroupBy implementation scheme which is widely used in distributed systems and databases. The GroupBy operation partitions a set of out-of-order records into groups. Due to the massive data size, many I/O-efficient grouping schemes that exploit external memory have been proposed. In this paper, we observe that the group sizes of many real data exhibit power-law property and the grouping schemes’ performance varies a lot for data with different group sizes. The indexing–filling approach prefers data with big group size, while the partitioned hash approach prefers data with small group size. Based on this observation, we propose a hybrid approach, PowerHash, which invokes different grouping schemes for different data. The group size information is approximately estimated by the count-min sketch so that the big groups and small groups can be distinguished from each other. With a given memory budget, our results show that PowerHash can improve performance by up to six times over the existing GroupBy implementations.

Lightweight Indexing and Querying Services for Big Spatial Data

With the widespread use of GPS-equipped smartphones and Internet of Things devices, a huge amount of data with location information is being generated at an unprecedented rate. To gain a deeper insight into such a plethora of spatial data, scientists and engineers are widely using spatial queries for their big data applications. However, because of not only the massive spatial data size but also the complexity of spatial query processing, they are struggling to efficiently process the spatial queries. In this paper, we propose lightweight and scalable indexing and querying services for big spatial data stored in distributed storage systems or graph-based systems. Our spatial services have several advantages over existing approaches. First, our services can be easily applied to existing storage systems or graph-based models without modifying the internal implementation of existing systems/models. Second, our services achieve high pruning power by efficiently selecting only relevant spatial objects based on a simple yet effective filter. Third, our services support a customizable and easy-to-use control of index data size by adjusting the precision of indexed geometries. Lastly, our services support efficient updates of spatial data. Our experimental results using real-world datasets validate the effectiveness and efficiency of our spatial services.

Cascaded Neural Network Based Data Mining Strategy for Cloud Intrusion Detection

In recent years data mining has acquired huge popularity in the field of knowledge discovery. Thus, this approach has inspired several researches for anomaly detection, fraud detection and intrusion detection with higher accuracy, all round generalization of the problem and its sub cases; all giving higher performance in conditions subjected to continuous alteration. Though there remain quite a few challenging problems in design and implementation of a data mining based cloud intrusion detection system, as deception tactics and modeling of behavior remains a daunting problem to compute for anomaly owing to massive size of data to process in reasonable time. In this study we present a cascaded neural network based data mining strategy for cloud intrusion detection systems (IDSs) and presents the comparison and performance results tested on DARPA Intrusion Detection (ID) Data Sets, Knowledge Discovery and Data Mining Cup, NSL-KDD dataset. The study exhibits numerous advantages offered by the presented method and give reliable results of anomaly detection in real time scenario.

Uncertainty quantification for radio interferometric imaging: II. MAP estimation

Uncertainty quantification is a critical missing component in radio interferometric imaging that will only become increasingly important as the big-data era of radio interferometry emerges. Statistical sampling approaches to perform Bayesian inference, like Markov Chain Monte Carlo (MCMC) sampling, can in principle recover the full posterior distribution of the image, from which uncertainties can then be quantified. However, for massive data sizes, like those anticipated from the Square Kilometre Array (SKA), it will be difficult if not impossible to apply any MCMC technique due to its inherent computational cost. We formulate Bayesian inference problems with sparsity-promoting priors (motivated by compressive sensing), for which we recover maximum a posteriori (MAP) point estimators of radio interferometric images by convex optimisation. Exploiting recent developments in the theory of probability concentration, we quantify uncertainties by post-processing the recovered MAP estimate. Three strategies to quantify uncertainties are developed: (i) highest posterior density credible regions; (ii) local credible intervals (cf. error bars) for individual pixels and superpixels; and (iii) hypothesis testing of image structure. These forms of uncertainty quantification provide rich information for analysing radio interferometric observations in a statistically robust manner. Our MAP-based methods are approximately $10^5$ times faster computationally than state-of-the-art MCMC methods and, in addition, support highly distributed and parallelised algorithmic structures. For the first time, our MAP-based techniques provide a means of quantifying uncertainties for radio interferometric imaging for realistic data volumes and practical use, and scale to the emerging big-data era of radio astronomy.

An adaptive lack of fit test for big data

abstractNew technological advancements combined with powerful computer hardware and high-speed network make big data available. The massive sample size of big data introduces unique computational challenges on scalability and storage of statistical methods. In this paper, we focus on the lack of fit test of parametric regression models under the framework of big data. We develop a computationally feasible testing approach via integrating the divide-and-conquer algorithm into a powerful nonparametric test statistic. Our theory results show that under mild conditions, the asymptotic null distribution of the proposed test is standard normal. Furthermore, the proposed test benefits from the use of data-driven bandwidth procedure and thus possesses certain adaptive property. Simulation studies show that the proposed method has satisfactory performances, and it is illustrated with an analysis of an airline data.

Big Data Characteristics, Challenges, Architectures, Analytics and Applications: A Review

Big data has evolved as a most challenging area in scientific study and research. It has drawn much attention during the last few years. It influences our modern society, business, government, healthcare, research and almost every discipline. In this data-driven era, where data is continuously acquired from a verity of sources for different purposes, the ability to make timely decisions based on available data is a very critical task. The massive data size, variety, velocity, accuracy and high dimensionality presents new challenges to big data. This paper attempts to present some challenges of big data. In addition, a study on the conceptual design of big data architecture presented on specific big data applications. In this paper, we presented research work on big data analytics techniques in the areas of text, sentiment, video, social media and predictive analytics. A comparative analysis on selected big data applications has been presented in detail.