Immersive Analytics Research Articles

Redefining Spectral Data Analysis with Immersive Analytics: Exploring Domain-Shifted Model Spaces for Optimal Model Selection.

Modern developments in autonomous chemometric machine learning technology strive to relinquish the need for human intervention. However, such algorithms developed and used in chemometric multivariate calibration and classification applications exclude crucial expert insight when difficult and safety-critical analysis situations arise, e.g., spectral-based medical decisions such as noninvasively determining if a biopsy is cancerous. The prediction accuracy and interpolation capabilities of autonomous methods for new samples depend on the quality and scope of their training (calibration) data. Specifically, analysis patterns within target data not captured by the training data will produce undesirable outcomes. Alternatively, using an immersive analytic approach allows insertion of human expert judgment at key machine learning algorithm junctures forming a sensemaking process performed in cooperation with a computer. The capacity of immersive virtual reality (IVR) environments to render human comprehensible three-dimensional space simulating real-world encounters, suggests its suitability as a hybrid immersive human-computer interface for data analysis tasks. Using IVR maximizes human senses to capitalize on our instinctual perception of the physical environment, thereby leveraging our innate ability to recognize patterns and visualize thresholds crucial to reducing erroneous outcomes. In this first use of IVR as an immersive analytic tool for spectral data, we examine an integrated IVR real-time model selection algorithm for a recent model updating method that adapts a model from the original calibration domain to predict samples from shifted target domains. Using near-infrared data, analyte prediction errors from IVR-selected models are reduced compared to errors using an established autonomous model selection approach. Results demonstrate the viability of IVR as a human data analysis interface for spectral data analysis including classification problems.

Investigating Professional Analyst Strategies in Immersive Space to Think.

Existing research on sensemaking in immersive analytics systems primarily focuses on understanding how users complete analysis within these systems with quantitative and qualitative datasets. However, these user studies mainly concentrate on understanding analysis styles and methodologies from a predominantly novice user study population. While this approach provides excellent initial insights into what users may do within IA systems, it fails to address how professionals may utilize an immersive analytic system for analysis tasks. In our work, we build upon an existing immersive analytics concept - "Immersive Space to Think" to understand how professional user populations differ from novice users in immersive analytic system usage. We conducted a user study with 11 professional intelligence analysts who completed three analysis sessions each. Using our results from this study, we provide deep analysis into how professional users complete sensemaking within immersive analytic systems, compare our findings to previously published findings with a novice user population, and provide insights into how to develop better IA systems to support the professional analyst's strategies within these systems.

Mint: Integrating scientific visualizations into virtual reality

We present an image-based approach to integrate state-of-the-art scientific visualization into virtual reality (VR) environments: the mint visualization/VR inter-operation system. We enable the integration of visualization algorithms from within their software frameworks directly into VR without the need to explicitly port visualization implementations to the underlying VR framework—thus retaining their capabilities, specializations, and optimizations. Consequently, our approach also facilitates enriching VR-based scientific data exploration with established or novel VR immersion and interaction techniques available in VR authoring tools. The separation of concerns enables researchers and users in different domains, like virtual immersive environments, immersive analytics, and scientific visualization, to independently work with existing software suitable for their domain while being able to interface with one another easily. We present our system architecture and inter-operation protocol (mint), an example of a collaborative VR environment implemented in the Unity engine (VRAUKE), as well as the integration of the protocol for the visualization frameworks Inviwo, MegaMol, and ParaView. Our implementation is publicly available as open-source software.Graphical abstract

Immersive analytics with augmented reality in meteorology: an exploratory study on ontology and linked data

Although Augmented Reality (AR) has been extensively studied in supporting Immersive Analytics (IA), there are still many challenges in visualising and interacting with big and complex datasets. To deal with these datasets, most AR applications utilise NoSQL databases for storing and querying data, especially for managing large volumes of unstructured or semi-structured data. However, NoSQL databases have limitations in their reasoning and inference capabilities, which can result in insufficient support for certain types of queries. To fill this gap, we aim to explore and evaluate whether an intelligent approach based on ontology and linked data can facilitate visual analytics tasks with big datasets on AR interface. We designed and implemented a prototype of this method for meteorological data analytics. An experiment was conducted to evaluate the use of a semantic database with linked data compared to a conventional approach in an AR-based immersive analytics system. The results significantly highlight the performance of semantic approach in helping the users analysing meteorological datasets and their subjective appreciation in working with the AR interface, which is enhanced with ontology and linked data.

A Systematic Literature Review of User Evaluation in Immersive Analytics

AbstractUser evaluation is a common and useful tool for systematically generating knowledge and validating novel approaches in the domain of Immersive Analytics. Since this research domain centres around users, user evaluation is of extraordinary relevance. Additionally, Immersive Analytics is an interdisciplinary field of research where different communities bring in their own methodologies. It is vital to investigate and synchronise these different approaches with the long‐term goal to reach a shared evaluation framework. While there have been several studies focusing on Immersive Analytics as a whole or on certain aspects of the domain, this is the first systematic review of the state of evaluation methodology in Immersive Analytics. The main objective of this systematic literature review is to illustrate methodologies and research areas that are still underrepresented in user studies by identifying current practice in user evaluation in the domain of Immersive Analytics in coherence with the PRISMA protocol. (see https://www.acm.org/publications/class-2012)

IMMERSIVE ANALYTICS

Immersive Analytics focuses on the benefits and challenges of using immersive environments for data analysis, and developing designs to improve efficiency. Although immersive technologies are widely available, practical solutions have not gained widespread acceptance in real- world applications. Research in this field focuses on abstract 3D visualization, immersive environments, paper sampling and use case evaluation. 03 Related Works • Brooks early review of VR applications found it effective in specific domains like flight simulators, automotive engineering, and astronaut training. • Van Dam et al. highlighted VR applications for SciVis, benefiting archaeology and medical fields. • Laha and Bowman reviewed VR techniques for visualizing volume data, highlighting the need for controlled experiments to explore individual components of immersion. • Reda et al. summarized research for hybrid reality environments like the CAVE2, emphasizing the possibility of collaborative data analysis. • Brath collected evidence that 3D visualizations offer advantages beyond 2D, focusing on immersive displays. 05 Result • Overall, immersive analytics can lead to better data comprehension, better decision-making, more engagement and teamwork, effective big data exploration, creative data visualization methods, non-technical users' empowerment, and applications in a variety of fields. • These results add to the increasing importance of immersive analytics as an efficient tool for decision support and data analysis. Introduction • To facilitate data-driven analytical reasoning, immersive analytics leverages interactive technology like as virtual reality glasses, big flat screen displays, and even the internet of things. • Immersion refers to an experience that creates a genuine sense of presence in a virtual environment. • A person experiences a shift in awareness from their immediate real environment to another reality. • Immersion analytics is still a relatively new field that has mostly been studied in use cases and workshops. • The notion remains extremely difficult, even within the small community of immersive analytics practitioners. Methodology 4.1 Immersive environments • The study focuses on immersive environments leading to a mixed reality experience. • Abstract 3D visualizations must be presented in a mixed or VR environment where hardware and user interact closely. 4.2 Abstract 3D visualizations • Data that lacks a physical representation or intrinsic spatial organization is referred to as abstract data. • In visualization, abstraction is obtained by using colors and shapes that are not directly associated with the object. 4.3 Paper sampling • Paper sampling in immersive analytics refers to the process of selecting and gathering relevant research papers and publications related to immersive analytics. Conclusion and future scope • In conclusion, immersive analytics has the potential to completely transform how we work with data by facilitating deeper understanding, better decision-making, and improved teamwork. • Immersion analytics will develop further and contribute significantly to data analysis and decision support in the future by tackling issues and seizing opportunities in technological innovation, domain-specific applications, ethical issues, and user experience design.

Is Native Naïve? Comparing Native Game Engines and WebXR as Immersive Analytics Development Platforms.

Native game engines have long been the 3-D development platform of choice for research in mixed and augmented reality. For this reason, they have also been adopted in many immersive visualization and immersive analytics systems and toolkits. However, with the rapid improvements of WebXR and related open technologies, this choice may not always be optimal for future visualization research. In this article, we investigate common assumptions about native game engines versus WebXR and find that while native engines still have an advantage in many areas, WebXR is rapidly catching up and is superior for many immersive analytics applications.

A Study on Collaborative Visual Data Analysis in Augmented Reality with Asymmetric Display Types.

Collaboration is a key aspect of immersive visual data analysis. Due to its inherent benefit of seeing co-located collaborators, augmented reality is often useful in such collaborative scenarios. However, to enable the augmentation of the real environment, there are different types of technology available. While there are constant developments in specific devices, each of these device types provide different premises for collaborative visual data analysis. In our work we combine handheld, optical see-through and video see-through displays to explore and understand the impact of these different device types in collaborative immersive analytics. We conducted a mixed-methods collaborative user study where groups of three performed a shared data analysis task in augmented reality with each user working on a different device, to explore differences in collaborative behaviour, user experience and usage patterns. Both quantitative and qualitative data revealed differences in user experience and usage patterns. For collaboration, the different display types influenced how well participants could participate in the collaborative data analysis, nevertheless, there was no measurable effect in verbal communication.

A New Generation of Collaborative Immersive Analytics on the Web: Open-Source Services to Capture, Process and Inspect Users’ Sessions in 3D Environments

Recording large amounts of users’ sessions performed through 3D applications may provide crucial insights into interaction patterns. Such data can be captured from interactive experiences in public exhibits, remote motion tracking equipment, immersive XR devices, lab installations or online web applications. Immersive analytics (IA) deals with the benefits and challenges of using immersive environments for data analysis and related design solutions to improve the quality and efficiency of the analysis process. Today, web technologies allow us to craft complex applications accessible through common browsers, and APIs like WebXR allow us to interact with and explore virtual 3D environments using immersive devices. These technologies can be used to access rich, immersive spaces but present new challenges related to performance, network bottlenecks and interface design. WebXR IA tools are still quite new in the literature: they present several challenges and leave quite unexplored the possibility of synchronous collaborative inspection. The opportunity to share the virtual space with remote analysts in fact improves sense-making tasks and offers new ways to discuss interaction patterns together, while inspecting captured records or data aggregates. Furthermore, with proper collaborative approaches, analysts are able to share machine learning (ML) pipelines and constructively discuss the outcomes and insights through tailored data visualization, directly inside immersive 3D spaces, using a web browser. Under the H2IOSC project, we present the first results of an open-source pipeline involving tools and services aimed at capturing, processing and inspecting interactive sessions collaboratively in WebXR with other analysts. The modular pipeline can be easily deployed in research infrastructures (RIs), remote dedicated hubs or local scenarios. The developed WebXR immersive analytics tool specifically offers advanced features for volumetric data inspection, query, annotation and discovery, alongside spatial interfaces. We assess the pipeline through users’ sessions captured during two remote public exhibits, by a WebXR application presenting generative AI content to visitors. We deployed the pipeline to assess the different services and to better understand how people interact with generative AI environments. The obtained results can be easily adopted for a multitude of case studies, interactive applications, remote equipment or online applications, to support or accelerate the detection of interaction patterns among remote analysts collaborating in the same 3D space.

Designing a 3D gestural interface to support user interaction with time-oriented data as immersive 3D radar charts

The design of intuitive three-dimensional user interfaces is vital for interaction in virtual reality, allowing to effectively close the loop between a human user and the virtual environment. The utilization of 3D gestural input allows for useful hand interaction with virtual content by directly grasping visible objects, or through invisible gestural commands that are associated with corresponding features in the immersive 3D space. The design of such interfaces remains complex and challenging. In this article, we present a design approach for a three-dimensional user interface using 3D gestural input with the aim to facilitate user interaction within the context of Immersive Analytics. Based on a scenario of exploring time-oriented data in immersive virtual reality using 3D Radar Charts, we implemented a rich set of features that is closely aligned with relevant 3D interaction techniques, data analysis tasks, and aspects of hand posture comfort. We conducted an empirical evaluation (n=12)\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$(n=12)$$\\end{document}, featuring a series of representative tasks to evaluate the developed user interface design prototype. The results, based on questionnaires, observations, and interviews, indicate good usability and an engaging user experience. We are able to reflect on the implemented hand-based grasping and gestural command techniques, identifying aspects for improvement in regard to hand detection and precision as well as emphasizing a prototype’s ability to infer user intent for better prevention of unintentional gestures.

Attention-Aware Visualization: Tracking and Responding to User Perception Over Time.

We propose the notion of attention-aware visualizations (AAVs) that track the user's perception of a visual representation over time and feed this information back to the visualization. Such context awareness is particularly useful for ubiquitous and immersive analytics where knowing which embedded visualizations the user is looking at can be used to make visualizations react appropriately to the user's attention: for example, by highlighting data the user has not yet seen. We can separate the approach into three components: (1) measuring the user's gaze on a visualization and its parts; (2) tracking the user's attention over time; and (3) reactively modifying the visual representation based on the current attention metric. In this paper, we present two separate implementations of AAV: a 2D data-agnostic method for web-based visualizations that can use an embodied eyetracker to capture the user's gaze, and a 3D data-aware one that uses the stencil buffer to track the visibility of each individual mark in a visualization. Both methods provide similar mechanisms for accumulating attention over time and changing the appearance of marks in response. We also present results from a qualitative evaluation studying visual feedback and triggering mechanisms for capturing and revisualizing attention.

VIRD: Immersive Match Video Analysis for High-Performance Badminton Coaching.

Badminton is a fast-paced sport that requires a strategic combination of spatial, temporal, and technical tactics. To gain a competitive edge at high-level competitions, badminton professionals frequently analyze match videos to gain insights and develop game strategies. However, the current process for analyzing matches is time-consuming and relies heavily on manual note-taking, due to the lack of automatic data collection and appropriate visualization tools. As a result, there is a gap in effectively analyzing matches and communicating insights among badminton coaches and players. This work proposes an end-to-end immersive match analysis pipeline designed in close collaboration with badminton professionals, including Olympic and national coaches and players. We present VIRD, a VR Bird (i.e., shuttle) immersive analysis tool, that supports interactive badminton game analysis in an immersive environment based on 3D reconstructed game views of the match video. We propose a top-down analytic workflow that allows users to seamlessly move from a high-level match overview to a detailed game view of individual rallies and shots, using situated 3D visualizations and video. We collect 3D spatial and dynamic shot data and player poses with computer vision models and visualize them in VR. Through immersive visualizations, coaches can interactively analyze situated spatial data (player positions, poses, and shot trajectories) with flexible viewpoints while navigating between shots and rallies effectively with embodied interaction. We evaluated the usefulness of VIRD with Olympic and national-level coaches and players in real matches. Results show that immersive analytics supports effective badminton match analysis with reduced context-switching costs and enhances spatial understanding with a high sense of presence.

Analyzing User Behaviour Patterns in a Cross-Virtuality Immersive Analytics System.

Recent work in immersive analytics suggests benefits for systems that support work across both 2D and 3D data visualizations, i.e., cross-virtuality analytics systems. Here, we introduce HybridAxes, an immersive visual analytics system that enables users to conduct their analysis either in 2D on desktop monitors or in 3D within an immersive AR environment - while enabling them to seamlessly switch and transfer their graphs between modes. Our user study results show that the cross-virtuality sub-systems in HybridAxes complement each other well in helping the users in their data-understanding journey. We show that users preferred using the AR component for exploring the data, while they used the desktop to work on more detail-intensive tasks. Despite encountering some minor challenges in switching between the two virtuality modes, users consistently rated the whole system as highly engaging, user-friendly, and helpful in streamlining their analytics processes. Finally, we present suggestions for designers of cross-virtuality visual analytics systems and identify avenues for future work.

Visual Cue Effects on a Classification Accuracy Estimation Task in Immersive Scatterplots.

Immersive visualization in virtual reality (VR) allows us to exploit visual cues for perception in 3D space, yet few existing studies have measured the effects of visual cues. Across a desktop monitor and a head-mounted display (HMD), we assessed scatterplot designs which vary their use of visual cues-motion, shading, perspective (graphical projection), and dimensionality-on two sets of data. We conducted a user study with a summary task in which 32 participants estimated the classification accuracy of an artificial neural network from the scatterplots. With Bayesian multilevel modeling, we capture the intricate visual effects and find that no cue alone explains all the variance in estimation error. Visual motion cues generally reduce participants' estimation error; besides this motion, using other cues may increase participants' estimation error. Using an HMD, adding visual motion cues, providing a third data dimension, or showing a more complicated dataset leads to longer response times. We speculate that most visual cues may not strongly affect perception in immersive analytics unless they change people's mental model about data. In summary, by studying participants as they interpret the output from a complicated machine learning model, we advance our understanding of how to use the visual cues in immersive analytics.

MD-Cave: An Immersive Visualization Workbench for Radiologists.

The MD-Cave is an immersive analytics system that provides enhanced stereoscopic visualizations to support visual diagnoses performed by radiologists. The system harnesses contemporary paradigms in immersive visualization and 3D interaction, which are better suited for investigating 3D volumetric data. We retain practicality through efficient utilization of desk space and comfort for radiologists in terms of frequent long duration use. MD-Cave is general and incorporates: (1) high resolution stereoscopic visualizations through a surround triple-monitor setup, (2) 3D interactions through head and hand tracking, (3) and a general framework that supports 3D visualization of deep-seated anatomical structures without the need for explicit segmentation algorithms. Such a general framework expands the utility of our system to many diagnostic scenarios. We have developed MD-Cave through close collaboration and feedback from two expert radiologists who evaluated the utility of MD-Cave and the 3D interactions in the context of radiological examinations. We also provide evaluation of MD-Cave through case studies performed by an expert radiologist and concrete examples on multiple real-world diagnostic scenarios, such as pancreatic cancer, shoulder-CT, and COVID-19 Chest CT examination.

Exploring the Evolution of Sensemaking Strategies in Immersive Space to Think.

Existing research on immersive analytics to support the sensemaking process focuses on single-session sensemaking tasks. However, in the wild, sensemaking can take days or months to complete. In order to understand the full benefits of immersive analytic systems, we need to understand how immersive analytic systems provide flexibility for the dynamic nature of the sensemaking process. In our work, we build upon an existing immersive analytic system - Immersive Space to Think, to evaluate how immersive analytic systems can support sensemaking tasks over time. We conducted a user study with eight participants with three separate analysis sessions each. We found significant differences between analysis strategies between sessions one, two, and three, which suggest that immersive space to think can benefit analysts during multiple stages in the sensemaking process.

Embodied Provenance for Immersive Sensemaking

Immersive analytics research has explored how embodied data representations and interactions can be used to engage users in sensemaking. Prior research has broadly overlooked the potential of immersive space for supporting analytic provenance, the understanding of sensemaking processes through users’ interaction histories. We propose the concept of embodied provenance, the use of three-dimensional space and embodied interactions in supporting recalling, reproducing, annotating and sharing analysis history in immersive environments. We design a conceptual framework for embodied provenance by highlighting a set of design criteria for analytic provenance drawn from prior work and identifying essential properties for embodied provenance. We develop a prototype system in virtual reality to demonstrate the concept and support the conceptual framework by providing multiple data views and embodied interaction metaphors in a large virtual space. We present a use case scenario of energy consumption analysis and evaluated the system through a qualitative evaluation with 17 participants, which show the system’s potential for assisting analytic provenance using embodiment. Our exploration of embodied provenance through this prototype provides lessons learnt to guide the design of immersive analytic tools for embodied provenance.

Immersive Unit Visualization with Augmented Reality

Immersive Unit Visualization is an emergent form of visualization that arose from Immersive Analytics where, unlike traditional visualizations, each data point is represented by an individual visual mark in an immersive virtual environment. This practice has focused almost exclusively on virtual reality, excluding augmented reality (AR). This article develops and tests a prototype of an Immersive Unit Visualization (Floating Companies II) with two AR devices: head-mounted display (HMD) and hand-held display (HHD). Results from the testing sessions with 20 users were analyzed through qualitative research analysis and thematic coding indicating that, while the HHD enabled a first contact with AR visualization on a familiar device, HMD improved the perception of hybrid space by supporting greater stability of virtual content, wider field of view, improved spatial perception, increased sense of immersion, and more realistic simulation, which had an impact on information reading and sense-making. The materialization of abstract quantitative values into concrete reality through its simulation in the real environment and the ludic dimension stand out as important opportunities for this type of visualization. This paper investigates the aspects distinguishing two experiences regarding data visualization in hybrid space, and characterizes ways of seeing information with AR, identifying opportunities to advance information design research.

X-Space: Interaction design of extending mixed reality space from Web2D visualization

Mixed reality offers a larger visualization space and more intuitive means of interaction for data exploration, and many works have been dedicated to combining 2D visualizations on screen with mixe reality. However, for each combination, we need to customize the implementation of the corresponding mixed reality 3D visualization. It is a challenge to simplify this development process and enable agile building of mixed reality 3D visualizations for 2D visualizations. In addition, many existing 2D visualizations do not provide interfaces oriented to immersive analytics, so how to extend the mixed reality 3D space from existing 2D visualizations is another challenge. This work presents an agile and flexible approach to interactively transfer visualizations from 2D screens to mixed reality 3D spaces. We designed an interactive process for spatial generation of mixed-reality 3D visualizations, defined a unified data transfer framework, integrated data deconstruction techniques for 2D visualizations, implemented interfaces to immersive visualization building tool-kits, and encapsulated these techniques into a tool named X-Space. We validated that the approach is feasible and effective through 2D visualization cases including scatter plots, stacked bar charts, and adjacency matrix. Finally, we conducted expert interviews to discuss the usability and value of the method.

IVMS: An immersive virtual meteorological sandbox based on WYSIWYG

A novel approach to visually represent meteorological data has emerged with the maturation of Immersive Analytics (IA). We have proposed an immersive meteorological virtual sandbox as a solution to the limitations of 2D analysis in expressing and perceiving data. This innovative visual method enables users to interact directly with data through non-contact aerial gestures (NCAG). Referring to the “What you see is what you get” concept in scientific visualization, we proposed a novel approach for the visual exploration of meteorological data that aims to immerse users in the analysis process. We hope this approach can inspire immersive visualization techniques for other types of geographic data as well. Finally, we conducted a user questionnaire to evaluate our system and work. The evaluation results demonstrate that our system effectively reduces cognitive burden, alleviates mental workload, and enhances users’ retention of analysis findings.