Large-scale Data Visualization Research Articles

Utilizing serverless framework for dynamic visualization and operations in geospatial applications

ABSTRACT While substantial efforts have been invested in the development of Discrete Global Grid Systems (DGGS) spatial operations and their potential applications in the geospatial domain, it has become evident that there is a demand for an efficient and scalable system to handle the visualization of large-scale DGGS data. This study demonstrated the potential of DGGS in conjunction with the serverless framework for dynamic visualization at various resolutions, which is based on data storage and effective querying using PostgreSQL integrated into Amazon Aurora Serverless. The use of Amazon Web Services (AWS) Lambda for on-the-fly generation of hexagon geometries significantly reduced the storage requirements and improved the speed of the visualization process. In addition, we implemented on-the-fly spatial operations including point binning, thresholding, aggregation, and neighborhood operations in the DGGS, highlighting the capabilities of DGGS in vector and raster processing. The proposed system has shown promising results in terms of efficiency, scalability, and adaptability, making it a viable solution for large-scale geospatial data processing and visualization. Case studies using flood risk data and terrain data further illustrate the system’s practical applicability in on-the-fly spatial operations and rapid visualization.

An integrated cyberinfrastructure system for water quality resources in the Upper Mississippi River Basin

ABSTRACT The Upper Mississippi Information System (UMIS) is a cyberinfrastructure framework designed to support large-scale real-time water-quality data integration, analysis, and visualization for the Upper Mississippi River Basin (UMRB). UMIS is intended to directly address three of the Grand Challenges for Engineering including (1) understanding access to clean drinking water, (2) management of the nitrogen cycle, and (3) engineering the tools of scientific discovery. The UMIS is designed to provide significant immediate and long-term impacts including a central platform for data access, integration, discovery, and adoption of cyberinfrastructure tools and services. The UMIS demonstrates that public data aggregators and central repositories can provide important services to anyone interested in water-quality research or education. In addition, working across multiple scales (e.g., state, region, county, or watershed) allows researchers to understand broad and narrow effects of water-quality strategies. Exploration of data across these scales encourages the development of problem-based research questions that can eventually provide feedback to public policies.

Interactive Volume Visualization via Multi-Resolution Hash Encoding Based Neural Representation.

Implicit neural networks have demonstrated immense potential in compressing volume data for visualization. However, despite their advantages, the high costs of training and inference have thus far limited their application to offline data processing and non-interactive rendering. In this article, we present a novel solution that leverages modern GPU tensor cores, a well-implemented CUDA machine learning framework, an optimized global-illumination-capable volume rendering algorithm, and a suitable acceleration data structure to enable real-time direct ray tracing of volumetric neural representations. Our approach produces high-fidelity neural representations with a peak signal-to-noise ratio (PSNR) exceeding 30 dB, while reducing their size by up to three orders of magnitude. Remarkably, we show that the entire training step can fit within a rendering loop, bypassing the need for pre-training. Additionally, we introduce an efficient out-of-core training strategy to support extreme-scale volume data, making it possible for our volumetric neural representation training to scale up to terascale on a workstation with an NVIDIA RTX 3090 GPU. Our method significantly outperforms state-of-the-art techniques in terms of training time, reconstruction quality, and rendering performance, making it an ideal choice for applications where fast and accurate visualization of large-scale volume data is paramount.

Towards Urban Digital Twins: A Workflow for Procedural Visualization Using Geospatial Data

A key feature for urban digital twins (DTs) is an automatically generated detailed 3D representation of the built and unbuilt environment from aerial imagery, footprints, LiDAR, or a fusion of these. Such 3D models have applications in architecture, civil engineering, urban planning, construction, real estate, Geographical Information Systems (GIS), and many other areas. While the visualization of large-scale data in conjunction with the generated 3D models is often a recurring and resource-intensive task, an automated workflow is complex, requiring many steps to achieve a high-quality visualization. Methods for building reconstruction approaches have come a long way, from previously manual approaches to semi-automatic or automatic approaches. This paper aims to complement existing methods of 3D building generation. First, we present a literature review covering different options for procedural context generation and visualization methods, focusing on workflows and data pipelines. Next, we present a semi-automated workflow that extends the building reconstruction pipeline to include procedural context generation using Python and Unreal Engine. Finally, we propose a workflow for integrating various types of large-scale urban analysis data for visualization. We conclude with a series of challenges faced in achieving such pipelines and the limitations of the current approach. However, the steps for a complete, end-to-end solution involve further developing robust systems for building detection, rooftop recognition, and geometry generation and importing and visualizing data in the same 3D environment, highlighting a need for further research and development in this field.

Large-scale spatial data visualization method based on augmented reality

BackgroundA task assigned to space exploration satellites involves detecting the physical environment within a certain space. However, space detection data are complex and abstract. These data are not conducive for researchers' visual perceptions of the evolution and interaction of events in the space environment. MethodsA time-series dynamic data sampling method for large-scale space was proposed for sample detection data in space and time, and the corresponding relationships between data location features and other attribute features were established. A tone-mapping method based on statistical histogram equalization was proposed and applied to the final attribute feature data. The visualization process is optimized for rendering by merging materials, reducing the number of patches, and performing other operations. ResultsThe results of sampling, feature extraction, and uniform visualization of the detection data of complex types, long duration spans, and uneven spatial distributions were obtained. The real-time visualization of large-scale spatial structures using augmented reality devices, particularly low-performance devices, was also investigated. ConclusionsThe proposed visualization system can reconstruct the three-dimensional structure of a large-scale space, express the structure and changes in the spatial environment using augmented reality, and assist in intuitively discovering spatial environmental events and evolutionary rules.

TooManyCellsInteractive: A visualization tool for dynamic exploration of single-cell data.

As single-cell sequencing technologies continue to advance, the growing volume and complexity of the ensuing data present new analytical challenges. Large cellular populations from single-cell atlases are more difficult to visualize and require extensive processing to identify biologically relevant subpopulations. Managing these workflows is also laborious for technical users and unintuitive for nontechnical users. We present TooManyCellsInteractive (TMCI), a browser-based JavaScript application for interactive exploration of cell populations. TMCI provides an intuitive interface to visualize and manipulate a radial tree representation of hierarchical cell subpopulations and allows users to easily overlay, filter, and compare biological features at multiple resolutions. Here we describe the software architecture and demonstrate how we used TMCI in a pan-cancer analysis to identify unique survival pathways among drug-tolerant persister cells. TMCI will facilitate exploration and visualization of large-scale sequencing data in a user-friendly way. TMCI is freely available at https://github.com/schwartzlab-methods/too-many-cells-interactive. An example tree from data within this article is available at https://tmci.schwartzlab.ca/.

DPNet: Dual-Path Network for Real-Time Object Detection With Lightweight Attention.

The recent advances in compressing high-accuracy convolutional neural networks (CNNs) have witnessed remarkable progress in real-time object detection. To accelerate detection speed, lightweight detectors always have few convolution layers using a single-path backbone. Single-path architecture, however, involves continuous pooling and downsampling operations, always resulting in coarse and inaccurate feature maps that are disadvantageous to locate objects. On the other hand, due to limited network capacity, recent lightweight networks are often weak in representing large-scale visual data. To address these problems, we present a dual-path network, named DPNet, with a lightweight attention scheme for real-time object detection. The dual-path architecture enables us to extract in parallel high-level semantic features and low-level object details. Although DPNet has a nearly duplicated shape with respect to single-path detectors, the computational costs and model size are not significantly increased. To enhance representation capability, a lightweight self-correlation module (LSCM) is designed to capture global interactions, with only a few computational overheads and network parameters. In the neck, LSCM is extended into a lightweight cross correlation module (LCCM), capturing mutual dependencies among neighboring scale features. We have conducted exhaustive experiments on MS COCO, Pascal VOC 2007, and ImageNet datasets. The experimental results demonstrate that DPNet achieves a state-of-the-art trade off between detection accuracy and implementation efficiency. More specifically, DPNet achieves 31.3% AP on MS COCO test-dev, 82.7% mAP on Pascal VOC 2007 test set, and 41.6% mAP on ImageNet validation set, together with nearly 2.5M model size, 1.04 GFLOPs, and 164 and 196 frames/s (FPS) FPS for [Formula: see text] input images of three datasets.

Adaptively Placed Multi-Grid Scene Representation Networks for Large-Scale Data Visualization.

Scene representation networks (SRNs) have been recently proposed for compression and visualization of scientific data. However, state-of-the-art SRNs do not adapt the allocation of available network parameters to the complex features found in scientific data, leading to a loss in reconstruction quality. We address this shortcoming with an adaptively placed multi-grid SRN (APMGSRN) and propose a domain decomposition training and inference technique for accelerated parallel training on multi-GPU systems. We also release an open-source neural volume rendering application that allows plug-and-play rendering with any PyTorch-based SRN. Our proposed APMGSRN architecture uses multiple spatially adaptive feature grids that learn where to be placed within the domain to dynamically allocate more neural network resources where error is high in the volume, improving state-of-the-art reconstruction accuracy of SRNs for scientific data without requiring expensive octree refining, pruning, and traversal like previous adaptive models. In our domain decomposition approach for representing large-scale data, we train an set of APMGSRNs in parallel on separate bricks of the volume to reduce training time while avoiding overhead necessary for an out-of-core solution for volumes too large to fit in GPU memory. After training, the lightweight SRNs are used for realtime neural volume rendering in our open-source renderer, where arbitrary view angles and transfer functions can be explored. A copy of this paper, all code, all models used in our experiments, and all supplemental materials and videos are available at https://github.com/skywolf829/APMGSRN.

Curating Training Data for Reliable Large-Scale Visual Data Analysis: Lessons from Identifying Trash in Street View Imagery

Visual data have dramatically increased in quantity in the digital age, presenting new opportunities for social science research. However, the extensive time and labor costs to process and analyze these data with existing approaches limit their use. Computer vision methods hold promise but often require large and nonexistent training data to identify sociologically relevant variables. We present a cost-efficient method for curating training data that utilizes simple tasks and pairwise comparisons to interpret and analyze visual data at scale using computer vision. We apply our approach to the detection of trash levels across space and over time in millions of street-level images in three physically distinct US cities. By comparing to ratings produced in a controlled setting and utilizing computational methods, we demonstrate generally high reliability in the method and identify sources that limit it. Altogether, this approach expands how visual data can be used at a large scale in sociology.

TissUUmaps 3: Improvements in interactive visualization, exploration, and quality assessment of large-scale spatial omics data

Background and objectivesSpatially resolved techniques for exploring the molecular landscape of tissue samples, such as spatial transcriptomics, often result in millions of data points and images too large to view on a regular desktop computer, limiting the possibilities in visual interactive data exploration. TissUUmaps is a free, open-source browser-based tool for GPU-accelerated visualization and interactive exploration of 107+ data points overlaying tissue samples. MethodsHerein we describe how TissUUmaps 3 provides instant multiresolution image viewing and can be customized, shared, and also integrated into Jupyter Notebooks. We introduce new modules where users can visualize markers and regions, explore spatial statistics, perform quantitative analyses of tissue morphology, and assess the quality of decoding in situ transcriptomics data. ResultsWe show that thanks to targeted optimizations the time and cost associated with interactive data exploration were reduced, enabling TissUUmaps 3 to handle the scale of today's spatial transcriptomics methods. ConclusionTissUUmaps 3 provides significantly improved performance for large multiplex datasets as compared to previous versions. We envision TissUUmaps to contribute to broader dissemination and flexible sharing of largescale spatial omics data.

Research on Visualization of Large-scale User Association Feature Data Based on Nonlinear Dimension Reduction Method

A high dimensional data visualization platform based on nonlinear dimension reduction approach was built and deployed in order to research the visualization of large-scale user linked feature data. The following test results were obtained through the implementation of a dimension reduction method and a functional module: The test set of the MNIST data set is given in CSV format, which may be represented as a 785*10000 matrix. The matrix is a representation of the handwritten picture that has been abstracted and transformed. The PCA approach provides the best dimensional-reduction impact on the dietary nutrient dataset, retaining 98.8 percent of the variation. The protein structure and function data set is not well served by any of the three dimensional-reduction techniques. Both T-SNE and Large Vis algorithms have better dimensional-reduction effects on MNIST data set, which reflects the nonlinear characteristics of the data set. Compared with T-SNE algorithm, Large Vis algorithm has no significant improvement in visualization effect, which is mainly reflected in time efficiency.

Technology Trends and Challenges for Large-Scale Scientific Visualization.

Scientific visualization is a key approach to understanding the growing massive streams of data from scientific simulations and experiments. In this article, I review technology trends including the positive effects of Moore's law on science, the significant gap between processing and data storage speeds, the emergence of hardware accelerators for ray-tracing, and the availability of robust machine learning techniques. These trends represent changes to the status quo and present the scientific visualization community with a new set of challenges. A major challenge involves extending our approaches to visualize the modern scientific process, which includes scientific verification and validation. Another key challenge to the community is the growing number, size, and complexity of scientific datasets. A final challenge is to take advantage of emerging technology trends in custom hardware and machine learning to significantly improve the large-scale data visualization process.

Key Technologies of Media Big Data in-Depth Analysis System Based on 5G Platform

To meet the needs of large-scale users for personalized streaming media services with high speed, low delay, and high quality in a 5G mobile network environment, this paper studies the resource allocation mechanism of streaming media based on a 5G network from the perspective of user demand prediction, which can alleviate the pressure of mobile network, improve the utilization rate of streaming media resources and the quality of user service experience. The augmented reality visualization of large-scale social media data must rely on the computing power of distributed clusters. This paper constructs a distributed parallel processing framework in a high-performance cluster environment, which adopts a loosely coupled organizational structure. Each module can be combined, called, and expanded arbitrarily under the condition of following a unified interface. In this paper, the algebraic method of parallel computing algorithm is innovatively proposed to describe parallel processing tasks and organize and call large-scale data-parallel processing operators, which effectively supports the business requirements of large-scale parallel processing of large-scale spatial social media data and solves the bottleneck of large-scale spatial social media data-parallel processing.

Soundscapes of urban parks: An innovative approach for ecosystem monitoring and adaptive management

Urban foresters are addressing the challenge of urban biodiversity loss through management plans in the context of rapid urbanization. Protecting the integrity of the urban ecosystem requires long-term monitoring and planning for resilience as well as effective management. The soundscape assessment has attracted attention in this field, but applying the soundscape assessment in urban ecological monitoring requires a protocol that links soundscapes to the impact of resource management on biodiversity over time. The effective processing and visualization of large-scale data also remains an important challenge. The aim of this study was to better understand the relationship between soundscape and physical environment, and examine the feasibility of this innovative soundscape approach in highly urbanized areas. Soundscape recordings were collected for 20 urban parks twice on 4 consecutive days in Spring. A total of 691,200 min of sound material were automatically obtained. In order to track the spatio-temporal patterns of a soundscape and determine its potential suitability for ecosystem monitoring, our study characterized soundscape information by adopting 4 widely used acoustic indices: acoustic diversity index (ADI), bioacoustic index (BIO), normalized difference vegetation index (NDSI), and power spectral density (PSD). Daily patterns of PSD have provided a potential connection between soundscapes and bird songs, and 1–2 kHz presented a similar pattern that was linked to human activity. Through further modeling, we tested the relationship of soundscapes to physical environment characteristics. The results showed the importance of habitat vegetation structure for acoustic diversity. More vertical heterogeneity, with an uneven canopy height or multilayered vegetation, was associated with more acoustic diversity. This suggests that clearing ground cover may have a significant negative impact on wildlife. Our results suggest that soundscape approaches provide a way to quickly synthesize large-scale recording data into meaningful patterns that can track changes in bird songs and ecosystem conditions. The proposed approach would enable regular assessment of urban parks and forests to inform adaptive planning and management strategies that can maintain or enhance biodiversity.

SLAM 3D Digital Terrain Mapping with SqueezeNet Driven by Road Traffic Data

In order to improve the efficiency of dynamic visualization of large-scale road traffic data in a web environment, this paper proposes a SLAM 3D digital terrain map visualization method using SqueezeNet in a web environment. We propose a hierarchical organization method of the road network, taking into account road attributes and drawing roads at different view heights; a multiroad merging method based on the line segment indexing feature of WebGL (web graphics library) technology, and optimizing the scene by combining view rejection and multithreading technology. The prototype system was developed and case studies were carried out using the national road network data as an example. The experimental results show that the frame rate of large-scale road traffic data visualization in the network environment is above 40 frames/second, which is 20–30 frames/second higher than that of Baidu’s ECharts GL visualization method.

Introducing the combined atlas framework for large-scale web-based data visualization: The GloNAF atlas of plant invasion.

Large‐scale biodiversity data, for example, on species distribution and richness information, are being mobilized and becoming available at an increasing rate. Interactive web applications like atlases have been developed to visualize available datasets and make them accessible to a wider audience. Web mapping tools are changing rapidly, and different underlying concepts have been developed to visualize datasets at a high cartographic standard.Here, we introduce the Combined Atlas Framework for the development of interactive web atlases for ecological data visualization. We combine two existing approaches: the five stages of the user‐centred design approach for web mapping applications and the three U approach for interface success.Subsequently, we illustrate the use of this framework by developing the Atlas of Plant Invasions based on the Global Naturalized Alien Flora (GloNAF) database. This case study illustrates how the newly developed Combined Atlas Framework with a user‐centred design philosophy can generate measurable success through communication with the target user group, iterative prototyping and competitive analysis of other existing web mapping approaches.The framework is useful in creating an atlas that employs user feedback to determine usability and utility features within an interactive atlas system. Finally, this framework will enable a better‐informed development process of future visualization and dissemination of biodiversity data through web mapping applications and interactive atlases.

HiIndex: An Efficient Spatial Index for Rapid Visualization of Large-Scale Geographic Vector Data

In the big data era, rapid visualization of large-scale vector data has become a serious challenge in Geographic Information Science (GIS). To fill the gap, we propose HiIndex, a spatial index that enables real-time and interactive visualization of large-scale vector data. HiIndex improves the state of the art with its low memory requirements, fast construction speed, and high visualization efficiency. In HiIndex, we present a tile-quadtree structure (TQ-tree) which divides the global geographic range based on the quadtree recursion method, and each node in the TQ-tree represents a specific and regular spatial range. In this paper, we propose a quick TQ-tree generation algorithm and an efficient visualization algorithm. Experiments show that the HiIndex is simple in structure, fast in construction, and less in memory occupation, and our approach can support interactive and real-time visualization of billion scale vector data with negligible pre-treatment time.

A Survey on OD Data Visualization

<p indent=0mm>OD data is a type of data that provides origin and destination positions, and sparsely describes the movement trajectory of objects. Being the research hotspots in data visualization and geospatial analysis, visual analysis of OD data contributes to discovering group movement patterns, exploring spatial anomalies and hidden relationships, and in-depth analysis of various statistics attributes. It is started from the pair-point structure of OD data in geospatial space, and classical OD data visualization methods and principles are elaborated, such as OD matrix, OD map, OD flow, etc. Then from the perspective of spatial-temporal feature exploration, spatial and spatial-temporal joint dimensions, the visual analysis of OD data is performed, while its application scenarios and enhanced significance of feature expression, perception, extraction and presentation are researched. To solve the problem of visual clutter in large-scale OD data visualization, the visual simplification methods such as filtering, aggregation, and sampling are sorted out in the research methods of visual analysis of OD data. Finally, the applications and specific cases of OD data visual analysis technology are investigated in different domains, the challenging problems in the research of OD data visual analysis are summarized, and the trend of its future development is prospected.

The Water–Energy–Food Nexus Discovery Map: Linking Geographic Information Systems, Academic Collaboration, and Large-Scale Data Visualization

The Water–Energy–Food (WEF) Nexus framework for holistic sustainable development has spawned independent and academic communities around the globe that utilize the framework in research, implementation, policy development, and technological advancement. These communities, however, are geographically and topically segmented and lack large-scale databasing that clearly catalogs and classifies their work. Recognizing this need, the WEF Nexus Strategic Initiative program at The Pennsylvania State University has developed the WEF Nexus Discovery Map utilizing the Arc Geographic Information Systems’ (GIS) Online Dashboard creation toolkit. In real time, users are able to select from 5040 different combinations of filters with the ease of a few button pushes and see projects pop up or disappear from the map located on the dashboard. Projects can then be clicked on to view their specific information, such as the institution that produced the work, local collaborators, relevant web page, and point of contact. The WEF Nexus Discovery Map demonstrates the early new-age of data resource management with the intersection of visuals, advanced search with built-in filters, and community-driven data collection to provide users with exact needs and connections to better facilitate and deploy the holistic sustainability framework of the WEF Nexus.

NetV.js: A web-based library for high-efficiency visualization of large-scale graphs and networks

Graph visualization plays an important role in several fields, such as social media networks, protein–protein interaction networks, and traffic networks. A number of visualization design tools and programming toolkits have been widely used in graph-related applications. However, a key challenge remains in the high-efficiency visualization of large-scale graph data. In this study, we present NetV.js, an open-source and WebGL-based JavaScript library that supports the fast visualization of large-scale graph data (up to 50 thousand nodes and 1 million edges) at an interactive frame rate with a commodity computer. Experimental results demonstrate that our library outperforms existing toolkits (Sigma.js, D3.js, Cytoscape.js, and Stardust.js) in terms of performance.