Immersive Analytics Research Articles

Deep Learning-based Computer Vision Algorithms, Immersive Analytics and Simulation Software, and Virtual Reality Modeling Tools in Digital Twin-driven Smart Manufacturing

Deep Learning-based Computer Vision Algorithms, Immersive Analytics and Simulation Software, and Virtual Reality Modeling Tools in Digital Twin-driven Smart Manufacturing

RoboHapalytics: A Robot Assisted Haptic Controller for Immersive Analytics.

Immersive environments offer new possibilities for exploring three-dimensional volumetric or abstract data. However, typical mid-air interaction offers little guidance to the user in interacting with the resulting visuals. Previous work has explored the use of haptic controls to give users tangible affordances for interacting with the data, but these controls have either: been limited in their range and resolution; were spatially fixed; or required users to manually align them with the data space. We explore the use of a robot arm with hand tracking to align tangible controls under the user's fingers as they reach out to interact with data affordances. We begin with a study evaluating the effectiveness of a robot-extended slider control compared to a large fixed physical slider and a purely virtual mid-air slider. We find that the robot slider has similar accuracy to the physical slider but is significantly more accurate than mid-air interaction. Further, the robot slider can be arbitrarily reoriented, opening up many new possibilities for tangible haptic interaction with immersive visualisations. We demonstrate these possibilities through three use-cases: selection in a time-series chart; interactive slicing of CT scans; and finally exploration of a scatter plot depicting time-varying socio-economic data.

Immersive Analytics with Abstract 3D Visualizations: A Survey

AbstractAfter a long period of scepticism, more and more publications describe basic research but also practical approaches to how abstract data can be presented in immersive environments for effective and efficient data understanding. Central aspects of this important research question in immersive analytics research are concerned with the use of 3D for visualization, the embedding in the immersive space, the combination with spatial data, suitable interaction paradigms and the evaluation of use cases. We provide a characterization that facilitates the comparison and categorization of published works and present a survey of publications that gives an overview of the state of the art, current trends, and gaps and challenges in current research.

Effect of Collaboration Mode and Position Arrangement on Immersive Analytics Tasks in Virtual Reality: A Pilot Study

[Background] Virtual reality (VR) technology can provide unique immersive experiences for group users, and especially for analytics tasks with visual information in learning. Providing a shared control/view may improve the task performance and enhance the user experience during VR collaboration. [Objectives] Therefore, this research explores the effect of collaborative modes and user position arrangements on task performance, user engagement, and collaboration behaviors and patterns in a VR learning environment that supports immersive collaborative tasks. [Method] The study involved two collaborative modes (shared and non-shared view and control) and three position arrangements (side-by-side, corner-to-corner, and back-to-back). A user study was conducted with 30 participants divided into three groups (Single, Shared, and Non-Shared) using a VR application that allowed users to explore the structural and transformational properties of 3D geometric shapes. [Results] The results showed that the shared mode would lead to higher task performance than single users for learning analytics tasks in VR. Besides, the side-by-side position got a higher score and more favor for enhancing the collaborative experience. [Conclusion] The shared view would be more suitable for improving task performance in collaborative VR. In addition, the side-by-side position may provide a higher user experience when collaborating in learning VR. From these results, a set of guidelines for the design of collaborative visualizations for VR environments are distilled and presented at the end of the paper. All in all, although our experiment is based on a colocated setting with two users, the results are applicable to both colocated and distributed collaborative scenarios with two or more users.

UrbanVR: An immersive analytics system for context-aware urban design

Urban design is a highly visual discipline that requires visualization for informed decision making. However, traditional urban design tools are mostly limited to representations on 2D displays that lack intuitive awareness. The popularity of head-mounted displays (HMDs) promotes a promising alternative with consumer-grade 3D displays. We introduce UrbanVR, an immersive analytics system with effective visualization and interaction techniques, to enable architects to assess designs in a virtual reality (VR) environment. Specifically, UrbanVR incorporates 1) a customized parallel coordinates plot (PCP) design to facilitate quantitative assessment of high-dimensional design metrics, 2) a series of egocentric interactions, including gesture interactions and handle-bar metaphors, to facilitate user interactions, and 3) a viewpoint optimization algorithm to help users explore both the PCP for quantitative analysis, and objects of interest for context awareness. Effectiveness and feasibility of the system are validated through quantitative user studies and qualitative expert feedbacks.

The Value of Immersive Visualization.

In recent years, research on immersive environments has experienced a new wave of interest, and immersive analytics has been established as a new research field. Every year, a vast amount of different techniques, applications, and user studies are published that focus on employing immersive environments for visualizing and analyzing data. Nevertheless, immersive analytics is still a relatively unexplored field that needs more basic research in many aspects and is still viewed with skepticism. Rightly so, because in our opinion, many researchers do not fully exploit the possibilities offered by immersive environments and, on the contrary, sometimes even overestimate the power of immersive visualizations. Although a growing body of papers has demonstrated individual advantages of immersive analytics for specific tasks and problems, the general benefit of using immersive environments for effective analytic tasks remains controversial. In this article, we reflect on when and how immersion may be appropriate for the analysis and present four guiding scenarios. We report on our experiences, discuss the landscape of assessment strategies, and point out the directions where we believe immersive visualizations have the greatest potential.

Comparing and Combining Virtual Hand and Virtual Ray Pointer Interactions for Data Manipulation in Immersive Analytics.

In this work, we evaluate two standard interaction techniques for Immersive Analytics environments: virtual hands, with actions such as grabbing and stretching, and virtual ray pointers, with actions assigned to controller buttons. We also consider a third option: seamlessly integrating both modes and allowing the user to alternate between them without explicit mode switches. Easy-to-use interaction with data visualizations in Virtual Reality enables analysts to intuitively query or filter the data, in addition to the benefit of multiple perspectives and stereoscopic 3D display. While many VR-based Immersive Analytics systems employ one of the studied interaction modes, the effect of this choice is unknown. Considering that each has different advantages, we compared the three conditions through a controlled user study in the spatio-temporal data domain. We did not find significant differences between hands and ray-casting in task performance, workload, or interactivity patterns. Yet, 60% of the participants preferred the mixed mode and benefited from it by choosing the best alternative for each low-level task. This mode significantly reduced completion times by 23% for the most demanding task, at the cost of a 5% decrease in overall success rates.

The Effect of Exploration Mode and Frame of Reference in Immersive Analytics.

The design space for user interfaces for Immersive Analytics applications is vast. Designers can combine navigation and manipulation to enable data exploration with ego- or exocentric views, have the user operate at different scales, or use different forms of navigation with varying levels of physical movement. This freedom results in a multitude of different viable approaches. Yet, there is no clear understanding of the advantages and disadvantages of each choice. Our goal is to investigate the affordances of several major design choices, to enable both application designers and users to make better decisions. In this article, we assess two main factors, exploration mode and frame of reference, consequently also varying visualization scale and physical movement demand. To isolate each factor, we implemented nine different conditions in a Space-Time Cube visualization use case and asked 36 participants to perform multiple tasks. We analyzed the results in terms of performance and qualitative measures and correlated them with participants' spatial abilities. While egocentric room-scale exploration significantly reduced mental workload, exocentric exploration improved performance in some tasks. Combining navigation and manipulation made tasks easier by reducing workload, temporal demand, and physical effort.

ShuttleSpace: Exploring and Analyzing Movement Trajectory in Immersive Visualization

We present ShuttleSpace, an immersive analytics system to assist experts in analyzing trajectory data in badminton. Trajectories in sports, such as the movement of players and balls, contain rich information on player behavior and thus have been widely analyzed by coaches and analysts to improve the players' performance. However, existing visual analytics systems often present the trajectories in court diagrams that are abstractions of reality, thereby causing difficulty for the experts to imagine the situation on the court and understand why the player acted in a certain way. With recent developments in immersive technologies, such as virtual reality (VR), experts gradually have the opportunity to see, feel, explore, and understand these 3D trajectories from the player's perspective. Yet, few research has studied how to support immersive analysis of sports data from such a perspective. Specific challenges are rooted in data presentation (e.g., how to seamlessly combine 2D and 3D visualizations) and interaction (e.g., how to naturally interact with data without keyboard and mouse) in VR. To address these challenges, we have worked closely with domain experts who have worked for a top national badminton team to design ShuttleSpace. Our system leverages 1) the peripheral vision to combine the 2D and 3D visualizations and 2) the VR controller to support natural interactions via a stroke metaphor. We demonstrate the effectiveness of ShuttleSpace through three case studies conducted by the experts with useful insights. We further conduct interviews with the experts whose feedback confirms that our first-person immersive analytics system is suitable and useful for analyzing badminton data.

Shared Surfaces and Spaces: Collaborative Data Visualisation in a Co-located Immersive Environment.

Immersive technologies offer new opportunities to support collaborative visual data analysis by providing each collaborator a personal, high-resolution view of a flexible shared visualisation space through a head mounted display. However, most prior studies of collaborative immersive analytics have focused on how groups interact with surface interfaces such as tabletops and wall displays. This paper reports on a study in which teams of three co-located participants are given flexible visualisation authoring tools to allow a great deal of control in how they structure their shared workspace. They do so using a prototype system we call FIESTA: the Free-roaming Immersive Environment to Support Team-based Analysis. Unlike traditional visualisation tools, FIESTA allows users to freely position authoring interfaces and visualisation artefacts anywhere in the virtual environment, either on virtual surfaces or suspended within the interaction space. Our participants solved visual analytics tasks on a multivariate data set, doing so individually and collaboratively by creating a large number of 2D and 3D visualisations. Their behaviours suggest that the usage of surfaces is coupled with the type of visualisation used, often using walls to organise 2D visualisations, but positioning 3D visualisations in the space around them. Outside of tightly-coupled collaboration, participants followed social protocols and did not interact with visualisations that did not belong to them even if outside of its owner's personal workspace.

The Data Visualisation and Immersive Analytics Research Lab at Monash University

This article reviews two decades of research in topics in Information Visualisation emerging from the Data Visualisation and Immersive Analytics Lab at Monash University Australia (Monash IA Lab). The lab has been influential with contributions in algorithms, interaction techniques and experimental results in Network Visualisation, Interactive Optimisation and Geographic and Cartographic visualisation. It has also been a leader in the emerging topic of Immersive Analytics, which explores natural interactions and immersive display technologies in support of data analytics. We reflect on advances in these areas but also sketch our vision for future research and developments in data visualisation more broadly.

Flex-ER

Extended Reality (XR) systems (which encapsulate AR, VR and MR) is an emerging field which enables the development of novel visualization and interaction techniques. To develop and to assess such techniques, researchers and designers have to face choices in terms of which development tools to adopt, and with very little information about how such tools support some of the very basic tasks for information visualization, such as selecting data items, linking and navigating. As a solution, we propose Flex-ER, a flexible web-based environment that enables users to prototype, debug and share experimental conditions and results. Flex-ER enables users to quickly switch between hardware platforms and input modalities by using a JSON specification that supports both defining interaction techniques and tasks at a low cost. We demonstrate the flexibility of the environment through three task design examples: brushing, linking and navigating. A qualitative user study suggest that Flex-ER can be helpful to prototype and explore different interaction techniques for immersive analytics.

Evaluating Usability Aspects of a Mixed Reality Solution for Immersive Analytics in Industry 4.0 Scenarios.

In medicine or industry, the analysis of high-dimensional data sets is increasingly required. However, available technical solutions are often complex to use. Therefore, new approaches like immersive analytics are welcome. Immersive analytics promise to experience high-dimensional data sets in a convenient manner for various user groups and data sets. Technically, virtual-reality devices are used to enable immersive analytics. In Industry 4.0, for example, scenarios like the identification of outliers or anomalies in high-dimensional data sets are pursued goals of immersive analytics. In this context, two important questions should be addressed for any developed technical solution on immersive analytics: First, is the technical solutions being helpful or not? Second, is the bodily experience of the technical solution positive or negative? The first question aims at the general feasibility of a technical solution, while the second one aims at the wearing comfort. Extant studies and protocols, which systematically address these questions are still rare. In this work, a study protocol is presented, which mainly investigates the usability for immersive analytics in Industry 4.0 scenarios. Specifically, the protocol is based on four pillars. First, it categorizes users based on previous experiences. Second, tasks are presented, which can be used to evaluate the feasibility of the technical solution. Third, measures are presented, which quantify the learning effect of a user. Fourth, a questionnaireevaluates the stress level when performing tasks. Based on these pillars, a technical setting was implemented that uses mixed reality smartglasses to apply the study protocol. The results of the conducted study show the applicability of the protocol on the one hand and the feasibility of immersive analytics in Industry 4.0 scenarios on the other. The presented protocol includes a discussion of discovered limitations.

Evaluating Usability Aspects of a Mixed Reality Solution for Immersive Analytics in Industry 4.0 Scenarios

Evaluating Usability Aspects of a Mixed Reality Solution for Immersive Analytics in Industry 4.0 Scenarios

EXPLORING CULTURAL HERITAGE COLLECTIONS IN IMMERSIVE ANALYTICS: CHALLENGES, BENEFITS, AND A CASE STUDY USING VIRTUAL REALITY

Abstract. Immersive analytics, at the intersection of visual analytics and virtual reality has recently gained some traction. Taking a similar approach, VaRt-DataExplorer project is concerned with exploration of data spaces in Virtual Reality (VR) in the context of cultural heritage collections. Our main objective is to facilitate better a understanding and insight into spatially referenced cultural heritage data sets. Within the scope of the project, this goal would be achieved by providing potentially ‘intuitive’ forms of real time interaction with the data, and rendering quickly recognizable visuospatial representations to offer more context to cultural artifacts. In particular, a spatial context is provided to the viewers by referencing geographical aspects of the data. Due to the incomplete and imprecise nature of data in this domain, thoughtful attention is given to visualization fidelity. Our initial user study suggests that using an immersive VR offers benefits for the exploration task for the viewer and the user experience provided by VaRt-DataExplorer has received high ratings.

VISUALIZING MULTILAYERED GEOSPATIAL DATA IN VIRTUAL REALITY TO ASSESS PUBLIC LIGHTING

Abstract. With the improvement and proliferation of virtual reality devices, their use for research and professional activity is broadening, fostering the advent of the field of immersive analytics, as is their acceptance among consumers. Other than the heightened sense of immersion into visualized data they provide, they also make displays of much larger apparent size and different positioning practical than what would be possible otherwise. Drawing on these benefits, we implemented a development of Multiple and Coordinated Displays (MCVs) for geovisualization that stacks different layers of data above each other, tilted for legibility. In a formal experiment, we evaluated it and two other, comparable MCV methods implemented in VR for their usefulness in analyzing public perception and soliciting public feedback regarding urban street lighting. In that field, the direction has recently been shifting from purely systemic development to a participatory approach, thus our investigation was into how a system like this could facilitate participation that can yield actionable results. Previous analysis of interaction data and usability questionnaires reveals preferences for certain systems depending on user characteristics, with the stack system showing a slight advantage over a grid of layers and especially over temporal multiplexing. We show that regardless of MCV variation, participants were able to analyze and provide feedback on public lighting situations that can directly contribute to urbanist work. The MCV approach further aided in understanding their choices, as eye-tracking allowed us to analyze attention to individual data layers.

Immersive diagnostics of reservoirs under WAG injection, part I – Understanding the dynamics around the injector

Advances in computational and immersive technologies are enabling one to draw new hypotheses and conclusions in diverse disciplines. This manuscript proposes several immersive diagnostic paradigms in reservoir engineering. These techniques provide means to understand complex dynamics: when, where, how different processes occur in the subsurface under diverse, realistic conditions. Through the lens of these immersive diagnostics, this study critically investigates reservoirs under WAG injection. Signatures of the proposed diagnostic measures are examined for reservoirs under WAG using four different injection-fluid streams (pure CO2, pure methane, lean gas, and enriched gas) and three WAG injection-cycle strategies in several depositional systems. Due to relatively high content and various scopes, this study appears in two parts. Part I of this study focused on learning the dynamics around the injector under diverse scenarios, whereas Part II explores the impact of reservoir heterogeneity.Immersive diagnostics provide a basis for reservoir tomography under various realistic subsurface and operational conditions. Immersive metrics presented in this study reveal many insightful observations. For example, they can shed light on the extent of the local miscibility in the reservoir settings, profiles of displacement efficiency for contrasting solvents, dissimilarity in the profiles of oil-in-place, and oil saturation. Contrarian views may also emerge using immersive analytics.The immersive analytics reveal phase trapping or banking behavior in oil-in-place, oil diffusivity, and gas-oil mobility ratio profiles around the injector. Dynamics around the injector and the producers can be very contrasting. CO2 and the enriched gas injection yields oil density reduction of 2–3 times, and viscosity reduction of about two times more profound into the reservoir, compared to C1 and lean-gas injection with the same injection strategy. Phase trapping or banking behavior appears in oil-in-place, oil diffusivity, and gas-oil mobility ratio profiles around the injector. Dynamics around the injector and the producers can be very contrasting.Late-time behavior of water-oil mobility ratio profiles indicates the compartmentalized nature of the reservoir dynamics, particularly when injecting CO2. For instance, a 5-month gas/1-month water injection strategy resulted in more significant variations in water-oil mobility and gas-oil mobility ratio. In contrast, a 5-month water/1-month gas injection showed less to no compartmentalization behavior. CO2 injection yielded a normal flow than transverse diffusive flow compared to the other injection fluids. The inertial component becomes dominant closer to the producers compared to the vicinity of the injector. The Péclet number signature indicates that much of the late-time displacement relates to diffusive flow than inertial flow.

AeroVR: An immersive visualisation system for aerospace design and digital twinning in virtual reality

ABSTRACTOne of today’s most propitious immersive technologies isvirtual reality(VR). This term is colloquially associated with headsets that transport users to a bespoke, built-for-purpose immersive 3D virtual environment. It has given rise to the field of immersive analytics—a new field of research that aims to use immersive technologies for enhancing and empowering data analytics. However, in developing such a new set of tools, one has to ask whether the move from standard hardware setup to a fully immersive 3D environment is justified—both in terms of efficiency and development costs. To this end, in this paper, we presentAeroVR—an immersive aerospace design environment with the objective of aiding the component aerodynamic design process by interactively visualizing performance and geometry. We decompose the design of such an environment into function structures, identify the primary and secondary tasks, present an implementation of the system, and verify the interface in terms of usability and expressiveness. We deploy AeroVR on a prototypical design study of a compressor blade for an engine.

VRIA: A Web-Based Framework for Creating Immersive Analytics Experiences.

We present VRIA, a Web-based framework for creating Immersive Analytics (IA) experiences in Virtual Reality. VRIA is built upon WebVR, A-Frame, React and D3.js, and offers a visualization creation workflow which enables users, of different levels of expertise, to rapidly develop Immersive Analytics experiences for the Web. The use of these open-standards Web-based technologies allows us to implement VR experiences in a browser and offers strong synergies with popular visualization libraries, through the HTML Document Object Model (DOM). This makes VRIA ubiquitous and platform-independent. Moreover, by using WebVR's progressive enhancement, the experiences VRIA creates are accessible on a plethora of devices. We elaborate on our motivation for focusing on open-standards Web technologies, present the VRIA creation workflow and detail the underlying mechanics of our framework. We also report on techniques and optimizations necessary for implementing Immersive Analytics experiences on the Web, discuss scalability implications of our framework, and present a series of use case applications to demonstrate the various features of VRIA. Finally, we discuss current limitations of our framework, the lessons learned from its development, and outline further extensions.

Task Dependent Group Coupling and Territorial Behavior on Large Tiled Displays.

Large display environments are highly suitable for immersive analytics. They provide enough space for effective co-located collaboration and allow users to immerse themselves in the data. To provide the best setting—in terms of visualization and interaction—for the collaborative analysis of a real-world task, we have to understand the group dynamics during the work on large displays. Among other things, we have to study, what effects different task conditions will have on user behavior. In this paper, we investigated the effects of task conditions on group behavior regarding collaborative coupling and territoriality during co-located collaboration on a wall-sized display. For that, we designed two tasks: a task that resembles the information foraging loop and a task that resembles the connecting facts activity. Both tasks represent essential sub-processes of the sensemaking process in visual analytics and cause distinct space/display usage conditions. The information foraging activity requires the user to work with individual data elements to look into details. Here, the users predominantly occupy only a small portion of the display. In contrast, the connecting facts activity requires the user to work with the entire information space. Therefore, the user has to overview the entire display. We observed 12 groups for an average of 2 h each and gathered qualitative data and quantitative data in the form of surveys, field notes, video recordings, tracking data, and system logs. During data analysis, we focused specifically on participants' collaborative coupling (in particular, collaboration tightness, coupling styles, user roles, and task subdivision strategies) and territorial behavior. Our results both confirm and extend findings from the previous tabletop and wall-sized display studies. We could detect that participants tended to subdivide the task to approach it, in their opinion, in a more effective way, in parallel. We describe the subdivision strategies for both task conditions. We also detected and described multiple user roles, as well as a new coupling style that does not fit in either category: loosely or tightly. Moreover, we could observe a territory type that has not been mentioned previously in research. In our opinion, this territory type can affect the collaboration process of groups with more than two collaborators negatively. Finally, we investigated critical display regions in terms of ergonomics. We could detect that users perceived some regions as less comfortable for long-time work. The findings can be valuable for groupware interface design and development of group behavior models for analytical reasoning and decision making.