Immersive Analytics Research Articles

Different realities: a comparison of augmented and virtual reality for the sensemaking process

Analysts perform sensemaking on large complex multimedia datasets in order to extract concepts, themes, and other kinds of insights from them. Immersive analytics, in particular, puts users in virtual environments that allow them to explore data in a unique way where they can interact and move through the data. Previous research using virtual reality immersive analytics tools found users wanting to refer to real-world objects or understand the physical world around them while continuing to perform their analysis. Therefore, we designed and ran a comparative study looking at the tradeoffs between virtual and augmented reality for our immersive analytics approach: Immersive Space to Think. Through two mixed-methods studies we found that virtual reality affords users a space where users can focus more on their task, but augmented reality allows them to use various real-world tools that can increase user satisfaction. In future immersive analytics tools, we recommend a blend of the two—augmented virtuality—with pass-through portals which allow users to see various real-world tools, such as whiteboards or desks and keyboards, while still giving themselves a space to focus.

RagRug: A Toolkit for Situated Analytics.

We present RagRug, an open-source toolkit for situated analytics. The abilities of RagRug go beyond previous immersive analytics toolkits by focusing on specific requirements emerging when using augmented reality (AR) rather than virtual reality. RagRug combines state of the art visual encoding capabilities with a comprehensive physical-virtual model, which lets application developers systematically describe the physical objects in the real world and their role in AR. We connect AR visualization with data streams from the Internet of Things using distributed dataflow. To this aim, we use reactive programming patterns so that visualizations become context-aware, i.e., they adapt to events coming in from the environment. The resulting authoring system is low-code; it emphasises describing the physical and the virtual world and the dataflow between the elements contained therein. We describe the technical design and implementation of RagRug, and report on five example applications illustrating the toolkit's abilities.

Beyond the horizon: immersive developments for animal ecology research

More diverse data on animal ecology are now available. This “data deluge” presents challenges for both biologists and computer scientists; however, it also creates opportunities to improve analysis and answer more holistic research questions. We aim to increase awareness of the current opportunity for interdisciplinary research between animal ecology researchers and computer scientists. Immersive analytics (IA) is an emerging research field in which investigations are performed into how immersive technologies, such as large display walls and virtual realityand augmented realitydevices, can be used to improve data analysis, outcomes, and communication. These investigations have the potential to reduce the analysis effort and widen the range of questions that can be addressed. We propose that biologists and computer scientists combine their efforts to lay the foundation for IA in animal ecology research. We discuss the potential and the challenges and outline a path toward a structured approach. We imagine that a joint effort would combine the strengths and expertise of both communities, leading to a well-defined research agenda and design space, practical guidelines, robust and reusable software frameworks, reduced analysis effort, and better comparability of results.

Answering Why and When?

Immersive analytics (IA) is a fast-growing research field that concerns improving and facilitating human sense making and data understanding through an immersive experience. Understanding the suitable application scenario that will benefit from IA enables a shift towards developing effective and meaningful applications. This paper aims to explore tasks and scenarios that can benefit from IA by conducting a systematic review of existing studies and mapping them according to the multi-level typology for abstract visualization tasks, which is also known as the what-why-how framework. The study synthesizes several works to answer the why within the context of multiple levels of specificity. In addition, this study also explores the application domains and IA guiding scenarios to address when scenarios best integrate with IA. Then, the paper discusses the IA evaluation types and research methods to evaluate an IA application that can promote effective user engagement in IA. Finally, the limitations and potential future works are discussed.

Cybercopters Swarm: Immersive analytics for alerts classification based on periodic data

This paper assesses the usefulness of an interactive and navigable 3D environment to help decision-making in cybersecurity. Malware programs frequently emit periodic signals in network logs; however, normal periodical network activities, such as software updates and data collection activities, mask them. Thus, if automatic systems use periodicity to successfully detect malware, they also detect ordinary activities as suspicious ones and raise false positives. Hence, there is a need to provide tools to sort the alerts raised by such software. Data visualizations can make it easier to categorize these alerts, as proven by previous research. However, traditional visualization tools can struggle to display a large amount of data that needs to be treated in cybersecurity in a clear way. In response, this paper explores the use of Immersive Analytics to interact with complex dataset representations and collect cues for alert classification. We created a prototype that uses a helical representation to underline periodicity in the distribution of one variable of a dataset. We tested this prototype in an alert triage scenario and compared it with a state-of-the-art 2D visualization with regard to the visualization efficiency, usability, workload, and flow induced.

Transforming graph data visualisations from 2D displays into augmented reality 3D space: A quantitative study

Modern video-based head-mounted displays allow users to operate along Milgram’s entire reality-virtuality continuum. This opens up the field for novel cross-reality applications that distribute data analytics tasks along this continuum to combine benefits of established 2D information visualisation in the real environment with immersive analytics. In this publication, we explore this potential by transforming 2D graph data from a planar, large-scale display in the real environment into a spherical layout in augmented reality 3D space, letting it appear as if the graph is moving out of the display. We focus on design aspects of this transformation that potentially help users to form a joint mental model of both visualisations and to continue their tasks seamlessly in augmented reality. For this purpose, we implemented a framework of transformation parameters that can be categorised as follows: transformation methods, node transformation order (groupings) and different ways of visual interconnection. Variants in each of these areas were investigated in three quantitative user studies in which users had to solve a simple cluster search task. We confirmed that a visual transformation from 2D to 3D helps users to continue their tasks in augmented reality with less interruptions, and that node transformation order should be adjusted to data and task context. We further identified that users can perform tasks more efficiently when a user-controlled transformation is used, while a constant transformation with fixed duration can contribute to lower error rates.

Labeling Out-of-View Objects in Immersive Analytics to Support Situated Visual Searching.

Augmented Reality (AR) embeds digital information into objects of the physical world. Data can be shown in-situ, thereby enabling real-time visual comparisons and object search in real-life user tasks, such as comparing products and looking up scores in a sports game. While there have been studies on designing AR interfaces for situated information retrieval, there has only been limited research on AR object labeling for visual search tasks in the spatial environment. In this article, we identify and categorize different design aspects in AR label design and report on a formal user study on labels for out-of-view objects to support visual search tasks in AR. We design three visualization techniques for out-of-view object labeling in AR, which respectively encode the relative physical position (height-encoded), the rotational direction (angle-encoded), and the label values (value-encoded) of the objects. We further implement two traditional in-view object labeling techniques, where labels are placed either next to the respective objects (situated) or at the edge of the AR FoV (boundary). We evaluate these five different label conditions in three visual search tasks for static objects. Our study shows that out-of-view object labels are beneficial when searching for objects outside the FoV, spatial orientation, and when comparing multiple spatially sparse objects. Angle-encoded labels with directional cues of the surrounding objects have the overall best performance with the highest user satisfaction. We discuss the implications of our findings for future immersive AR interface design.

Design Patterns for Situated Visualization in Augmented Reality.

Situated visualization has become an increasingly popular research area in the visualization community, fueled by advancements in augmented reality (AR) technology and immersive analytics. Visualizing data in spatial proximity to their physical referents affords new design opportunities and considerations not present in traditional visualization, which researchers are now beginning to explore. However, the AR research community has an extensive history of designing graphics that are displayed in highly physical contexts. In this work, we leverage the richness of AR research and apply it to situated visualization. We derive design patterns which summarize common approaches of visualizing data in situ. The design patterns are based on a survey of 293 papers published in the AR and visualization communities, as well as our own expertise. We discuss design dimensions that help to describe both our patterns and previous work in the literature. This discussion is accompanied by several guidelines which explain how to apply the patterns given the constraints imposed by the real world. We conclude by discussing future research directions that will help establish a complete understanding of the design of situated visualization, including the role of interactivity, tasks, and workflows.

Unraveling the Design Space of Immersive Analytics: A Systematic Review.

Immersive analytics has emerged as a promising research area, leveraging advances in immersive display technologies and techniques, such as virtual and augmented reality, to facilitate data exploration and decision-making. This paper presents a systematic literature review of 73 studies published between 2013-2022 on immersive analytics systems and visualizations, aiming to identify and categorize the primary dimensions influencing their design. We identified five key dimensions: Academic Theory and Contribution, Immersive Technology, Data, Spatial Presentation, and Visual Presentation. Academic Theory and Contribution assess the motivations behind the works and their theoretical frameworks. Immersive Technology examines the display and input modalities, while Data dimension focuses on dataset types and generation. Spatial Presentation discusses the environment, space, embodiment, and collaboration aspects in IA, and Visual Presentation explores the visual elements, facet and position, and manipulation of views. By examining each dimension individually and cross-referencing them, this review uncovers trends and relationships that help inform the design of immersive systems visualizations. This analysis provides valuable insights for researchers and practitioners, offering guidance in designing future immersive analytics systems and shaping the trajectory of this rapidly evolving field.

Interactive Extended Reality Techniques in Information Visualization

Immersive techniques, such as virtual reality, augmented reality, and mixed reality, take immersive displays as carriers to provide immersive experience. A large number of approaches focus on the visualization of scientific data in immersive environments while just a few methods concentrate on interactive information visualization (InfoVis) in an immersive environment, although InfoVis has been extended to the 3-D space for a long time. In the era of data explosion, the traditional 2-D space is unable to convey large amounts of abstract information in an intuitive way. Meanwhile, desktop-based 3-D InfoVis generally leads to visual conflict and confusion owing to limited display size and field of vision. In this survey, we search for the interactive techniques in immersive InfoVis and summarize their commonalities and discuss their differences and potential trends. The data types of abstract information in InfoVis can be categorized into graph/network data, high-dimensional and multivariate data, time-varying data, and text and document data. Besides, the visual presentation of information in immersive environments is also summarized, especially for charts, plots, and diagrams, which are some basic components of InfoVis techniques. We also described the immersive applications of InfoVis techniques, including the tools or frameworks on immersive analytics and infographics. The discussion about the traditional nonimmersive and the immersive methods in data visualizations show that the latter one has the potential to become an alternative to explore massive information in the future.

"Is It My Turn?"

Immersive analytics has the potential to promote collaboration in machine learning (ML). This is desired due to the specific characteristics of ML modeling in practice, namely the complexity of ML, the interdisciplinary approach in industry, and the need for ML interpretability. In this work, we introduce an augmented reality-based system for collaborative immersive analytics that is designed to support ML modeling in interdisciplinary teams. We conduct a user study to examine how collaboration unfolds when users with different professional backgrounds and levels of ML knowledge interact in solving different ML tasks. Specifically, we use the pair analytics methodology and performance assessments to assess collaboration and explore their interactions with each other and the system. Based on this, we provide qualitative and quantitative results on both teamwork and taskwork during collaboration. Our results show how our system elicits sustained collaboration as measured along six distinct dimensions. We finally make recommendations how immersive systems should be designed to elicit sustained collaboration in ML modeling.

Visual Comparison of Networks in VR.

Networks are an important means for the representation and analysis of data in a variety of research and application areas. While there are many efficient methods to create layouts for networks to support their visual analysis, approaches for the comparison of networks are still underexplored. Especially when it comes to the comparison of weighted networks, which is an important task in several areas, such as biology and biomedicine, there is a lack of efficient visualization approaches. With the availability of affordable high-quality virtual reality (VR) devices, such as head-mounted displays (HMDs), the research field of immersive analytics emerged and showed great potential for using the new technology for visual data exploration. However, the use of immersive technology for the comparison of networks is still underexplored. With this work, we explore how weighted networks can be visually compared in an immersive VR environment and investigate how visual representations can benefit from the extended 3D design space. For this purpose, we develop different encodings for 3D node-link diagrams supporting the visualization of two networks within a single representation and evaluate them in a pilot user study. We incorporate the results into a more extensive user study comparing node-link representations with matrix representations encoding two networks simultaneously. The data and tasks designed for our experiments are similar to those occurring in real-world scenarios. Our evaluation shows significantly better results for the node-link representations, which is contrary to comparable 2D experiments and indicates a high potential for using VR for the visual comparison of networks.

Towards a hybrid user interface for the visual exploration of large biomolecular networks using virtual reality.

Biomolecular networks, including genome-scale metabolic models (GSMMs), assemble the knowledge regarding the biological processes that happen inside specific organisms in a way that allows for analysis, simulation, and exploration. With the increasing availability of genome annotations and the development of powerful reconstruction tools, biomolecular networks continue to grow ever larger. While visual exploration can facilitate the understanding of such networks, the network sizes represent a major challenge for current visualisation systems. Building on promising results from the area of immersive analytics, which among others deals with the potential of immersive visualisation for data analysis, we present a concept for a hybrid user interface that combines a classical desktop environment with a virtual reality environment for the visual exploration of large biomolecular networks and corresponding data. We present system requirements and design considerations, describe a resulting concept, an envisioned technical realisation, and a systems biology usage scenario. Finally, we discuss remaining challenges.

Touching data with PropellerHand

Immersive analytics often takes place in virtual environments which promise the users immersion. To fulfill this promise, sensory feedback, such as haptics, is an important component, which is however not well supported yet. Existing haptic devices are often expensive, stationary, or occupy the user’s hand, preventing them from grasping objects or using a controller. We propose PropellerHand, an ungrounded hand-mounted haptic device with two rotatable propellers, that allows exerting forces on the hand without obstructing hand use. PropellerHand is able to simulate feedback such as weight and torque by generating thrust up to 11 N in 2-DOF and a torque of 1.87 Nm in 2-DOF. Its design builds on our experience from quantitative and qualitative experiments with different form factors and parts. We evaluated our prototype through a qualitative user study in various VR scenarios that required participants to manipulate virtual objects in different ways, while changing between torques and directional forces. Results show that PropellerHand improves users’ immersion in virtual reality. Additionally, we conducted a second user study in the field of immersive visualization to investigate the potential benefits of PropellerHand there.Graphical abstract

An Interactive Augmented Reality Graph Visualization for Chinese Painters

Recent research in the area of immersive analytics demonstrated the utility of augmented reality for data analysis. However, there is a lack of research on how to facilitate engaging, embodied, and interactive AR graph visualization. In this paper, we explored the design space for combining the capabilities of AR with node-link diagrams to create immersive data visualization. We first systematically described the design rationale and the design process of the mobile based AR graph including the layout, interactions, and aesthetics. Then, we validated the AR concept by conducting a user study with 36 participants to examine users’ behaviors with an AR graph and a 2D graph. The results of our study showed the feasibility of using an AR graph to present data relations and also introduced interaction challenges in terms of the effectiveness and usability with mobile devices. Third, we iterated the AR graph by implementing embodied interactions with hand gestures and addressing the connection between the physical objects and the digital graph. This study is the first step in our research, aiming to guide the design of the application of immersive AR data visualization in the future.

Virtual reality for the observation of oncology models (VROOM): immersive analytics for oncology patient cohorts

The significant advancement of inexpensive and portable virtual reality (VR) and augmented reality devices has re-energised the research in the immersive analytics field. The immersive environment is different from a traditional 2D display used to analyse 3D data as it provides a unified environment that supports immersion in a 3D scene, gestural interaction, haptic feedback and spatial audio. Genomic data analysis has been used in oncology to understand better the relationship between genetic profile, cancer type, and treatment option. This paper proposes a novel immersive analytics tool for cancer patient cohorts in a virtual reality environment, virtual reality to observe oncology data models. We utilise immersive technologies to analyse the gene expression and clinical data of a cohort of cancer patients. Various machine learning algorithms and visualisation methods have also been deployed in VR to enhance the data interrogation process. This is supported with established 2D visual analytics and graphical methods in bioinformatics, such as scatter plots, descriptive statistical information, linear regression, box plot and heatmap into our visualisation. Our approach allows the clinician to interrogate the information that is familiar and meaningful to them while providing them immersive analytics capabilities to make new discoveries toward personalised medicine.

FORSETI: A visual analysis environment enabling provenance awareness for the accountability of e-autopsy reports

Autopsy reports play a pivotal role in forensic science. Medical examiners (MEs) and diagnostic radiologists (DRs) cross-reference autopsy results in the form of autopsy reports, while judicial personnel derive legal documents from final autopsy reports. In our prior study, we presented a visual analysis system called the forensic autopsy system for e-court instruments (FORSETI) with an extended legal medicine markup language (x-LMML) that enables MEs and DRs to author and review e-autopsy reports. In this paper, we present our extended work to incorporate provenance infrastructure with authority management into FORSETI for forensic data accountability, which contains two features. The first is a novel provenance management mechanism that combines the forensic autopsy workflow management system (FAWfMS) and a version control system called lmmlgit for x-LMML files. This management mechanism allows much provenance data on e-autopsy reports and their documented autopsy processes to be individually parsed. The second is provenance-supported immersive analytics, which is intended to ensure that the DRs’ and MEs’ autopsy provenances can be viewed, listed, and analyzed so that a principal ME can author their own report through accountable autopsy referencing in an augmented reality setting. A fictitious case with a synthetic wounded body is used to demonstrate the effectiveness of the provenance-aware FORSETI system in terms of data accountability through the experience of experts in legal medicine.

Exploration and Assessment of Interaction in an Immersive Analytics Module: A Software-Based Comparison

The focus of computer systems in the field of visual analytics is to make the results clear and understandable. However, enhancing human-computer interaction (HCI) in the field is less investigated. Data visualization and visual analytics (VA) are usually performed using traditional desktop settings and mouse interaction. These methods are based on the window, icon, menu, and pointer (WIMP) interface, which often results in information clutter and is difficult to analyze and understand, especially by novice users. Researchers believe that introducing adequate, natural interaction techniques to the field is necessary for building effective and enjoyable visual analytics systems. This work introduces a novel virtual reality (VR) module to perform basic visual analytics tasks and aims to explore new interaction techniques in the field. A pilot study was conducted to measure the time it takes students to perform basic tasks for analytics using the developed VR module and compares it to the time it takes them to perform the same tasks using a traditional desktop to assess the effectiveness of the VR module in enhancing student’s performance. The results show that novice users (Participants with less programming experience) took about 50% less time to complete tasks using the developed VR module as a comrade to a programming language, notably R. Experts (Participants with advanced programming experience) took about the same time to complete tasks under both conditions (R and VR).

A Survey on Cross‐Virtuality Analytics

AbstractCross‐virtuality analytics (XVA) is a novel field of research within immersive analytics and visual analytics. A broad range of heterogeneous devices across the reality–virtuality continuum, along with respective visual metaphors and analysis techniques, are currently becoming available. The goal of XVA is to enable visual analytics that use transitional and collaborative interfaces to seamlessly integrate different devices and support multiple users. In this work, we take a closer look at XVA and analyse the existing body of work for an overview of its current state. We classify the related literature regarding ways of establishing cross‐virtuality by interconnecting different stages in the reality–virtuality continuum, as well as techniques for transitioning and collaborating between the different stages. We provide insights into visualization and interaction techniques employed in current XVA systems. We report on ways of evaluating such systems, and analyse the domains where such systems are becoming available. Finally, we discuss open challenges in XVA, giving directions for future research.

An Empirical Evaluation of Asymmetric Synchronous Collaboration Combining Immersive and Non-Immersive Interfaces Within the Context of Immersive Analytics

Collaboration is an essential part of data analysis, allowing multiple users to combine their expertise and to debate about the interpretation of data discoveries using their contextual knowledge. The design of collaborative interfaces within the context of Immersive Analytics remains challenging, particularly due to the various user-centered characteristics of immersive technologies. In this article, we present the use case of a system that enables multiple users to synchronously explore the same data in a collaborative scenario that combines immersive and non-immersive interfaces in an asymmetric role setup. Such a setup allows for bridging the gap when applying heterogeneous display and interaction technologies, enabling each analyst to have an independent and different view of the data, while maintaining important collaborative aspects during the joint data exploration. We developed an immersive VR environment (head-mounted display, 3D gestural input) and a non-immersive desktop terminal (monitor, keyboard and mouse) centered around spatio-temporal data exploration. Supported through a real-time communication interface, synchronous collaborative features are integrated in both interfaces, facilitating the users in their ability to establish a shared context and to make spatio-temporal references. We conducted an empirical evaluation with five participant pairs (within-subject design) to investigate aspects of usability, user engagement, and collaboration during a confirmative analysis task. Synthesis of questionnaire results in combination with additional log file analysis, audio activity analysis, and observations, revealed good usability scores, high user engagement, as well as overall close and balanced collaboration of enthusiastic pairs during the task completion independent of their interface type, validating our system approach in general. Further supported through the self-constructed Spatio-Temporal Collaboration Questionnaire, we are able to contribute with discussion and considerations of the presented scenario and the synchronous collaborative features for the design of similar applications.