3D Widgets Research Articles

Co-Designing Dynamic Mixed Reality Drill Positioning Widgets: A Collaborative Approach with Dentists in a Realistic Setup.

Mixed Reality (MR) is proven in the literature to support precise spatial dental drill positioning by superimposing 3D widgets. Despite this, the related knowledge about widget's visual design and interactive user feedback is still limited. Therefore, this study is contributed to by co-designed MR drill tool positioning widgets with two expert dentists and three MR experts. The results of co-design are two static widgets (SWs): a simple entry point, a target axis, and two dynamic widgets (DWs), variants of dynamic error visualization with and without a target axis (DWTA and DWEP). We evaluated the co-designed widgets in a virtual reality simulation supported by a realistic setup with a tracked phantom patient, a virtual magnifying loupe, and a dentist's foot pedal. The user study involved 35 dentists with various backgrounds and years of experience. The findings demonstrated significant results; DWs outperform SWs in positional and rotational precision, especially with younger generations and subjects with gaming experiences. The user preference remains for DWs (19) instead of SWs (16). However, findings indicated that the precision positively correlates with the time trade-off. The post-experience questionnaire (NASA-TLX) showed that DWs increase mental and physical demand, effort, and frustration more than SWs. Comparisons between DWEP and DWTA show that the DW's complexity level influences time, physical and mental demands. The DWs are extensible to diverse medical and industrial scenarios that demand precision.

Constraint-based bare-hand immersive 3D modelling

Abstract Three-dimensional user interfaces that are controlled by the user’s bare hands are mostly based on purely gesture-based interaction techniques. However, these interfaces are often slow and error prone. Especially in the field of immersive 3D modelling, gestures are unsuitable because they complicate and delay the modelling process. To address these problems, we present a new gesture-free 3D modelling technique called “3D touch-and-drag”, which allows users to select vertices by approaching them and to terminate operations by moving the 3D cursor (e.g. the forefinger) away from the constraint geometry (e.g. a straight line or a plane). Our proposed technique makes it possible to transfer the existing 3D modelling concepts (“3D widgets”) to virtual environments, as shown by an experimental 3D modelling tool. The gesture-free bare-hand interaction also improves the possibility of tactile feedback during 3D manipulation. We compared different modelling techniques for controlling the 3D widgets. We found that controller-based techniques are significantly faster than finger-tracking-based techniques. The 3D touch-and-drag technique is about as fast as gesture-based interactions. Mouse interaction in a two-dimensional GUI is only slightly faster than the 3D modelling techniques. Since our proposed technique has proven to be at least equivalent to gesture-based interaction techniques in terms of accuracy and efficiency, its further development using more accurate tracking techniques seems promising to exploit the advantages of hands-free and gesture-free interaction for immersive 3D modelling.

3D printing wireless connected objects

Our goal is to 3D print wireless sensors, input widgets and objects that can communicate with smartphones and other Wi-Fi devices, without the need for batteries or electronics. To this end, we present a novel toolkit for wireless connectivity that can be integrated with 3D digital models and fabricated using commodity desktop 3D printers and commercially available plastic filament materials. Specifically, we introduce the first computational designs that 1) send data to commercial RF receivers including Wi-Fi, enabling 3D printed wireless sensors and input widgets, and 2) embed data within objects using magnetic fields and decode the data using magnetometers on commodity smartphones. To demonstrate the potential of our techniques, we design the first fully 3D printed wireless sensors including a weight scale, flow sensor and anemometer that can transmit sensor data. Furthermore, we 3D print eyeglass frames, armbands as well as artistic models with embedded magnetic data. Finally, we present various 3D printed application prototypes including buttons, smart sliders and physical knobs that wirelessly control music volume and lights as well as smart bottles that can sense liquid flow and send data to nearby RF devices, without batteries or electronics.

An Open-Source Label Atlas Correction Tool and Preliminary Results on Huntingtons Disease Whole-Brain MRI Atlases.

The creation of high-quality medical imaging reference atlas datasets with consistent dense anatomical region labels is a challenging task. Reference atlases have many uses in medical image applications and are essential components of atlas-based segmentation tools commonly used for producing personalized anatomical measurements for individual subjects. The process of manual identification of anatomical regions by experts is regarded as a so-called gold standard; however, it is usually impractical because of the labor-intensive costs. Further, as the number of regions of interest increases, these manually created atlases often contain many small inconsistently labeled or disconnected regions that need to be identified and corrected. This project proposes an efficient process to drastically reduce the time necessary for manual revision in order to improve atlas label quality. We introduce the LabelAtlasEditor tool, a SimpleITK-based open-source label atlas correction tool distributed within the image visualization software 3D Slicer. LabelAtlasEditor incorporates several 3D Slicer widgets into one consistent interface and provides label-specific correction tools, allowing for rapid identification, navigation, and modification of the small, disconnected erroneous labels within an atlas. The technical details for the implementation and performance of LabelAtlasEditor are demonstrated using an application of improving a set of 20 Huntingtons Disease-specific multi-modal brain atlases. Additionally, we present the advantages and limitations of automatic atlas correction. After the correction of atlas inconsistencies and small, disconnected regions, the number of unidentified voxels for each dataset was reduced on average by 68.48%.

Volume-Rendering-Based Interactive 3D Measurement for Quantitative Analysis of 3D Medical Images

3D medical images are widely used to assist diagnosis and surgical planning in clinical applications, where quantitative measurement of interesting objects in the image is of great importance. Volume rendering is widely used for qualitative visualization of 3D medical images. In this paper, we introduce a volume-rendering-based interactive 3D measurement framework for quantitative analysis of 3D medical images. In the framework, 3D widgets and volume clipping are integrated with volume rendering. Specifically, 3D plane widgets are manipulated to clip the volume to expose interesting objects. 3D plane widgets, 3D line widgets, and 3D angle widgets are then manipulated to measure the areas, distances, and angles of interesting objects. The methodology of the proposed framework is described. Experimental results indicate the performance of the interactive 3D measurement framework.

Computer-Aided Detection and Diagnosis in Medical Imaging

Computer-Aided Detection and Diagnosis in Medical Imaging

Interacting with 3D Reactive Widgets for Musical Performance

While virtual reality and 3D interaction open new prospects for musical performance, existing immersive virtual instruments are often limited to single process instruments or musical navigation tools. We believe that immersive virtual environments may be used to design expressive and efficient multi-process instruments. In this paper we present the 3D reactive widgets. These graphical elements enable efficient and simultaneous control and visualization of musical processes. Then we describe Piivert, a novel input device that we have developed to manipulate these widgets, and several techniques for 3D musical interaction.

3D Interaction Design and Implementation in Virtual Spinal Fixation Surgery System

The 3D interaction design solution and implementation with VTK and QT were proposed in the system of virtual spinal fixation surgery. After discussing the background of 3D interaction technology, the principles of the interaction design about 3D interaction used in the system was given. According to the requirement of the system, the observer pattern and command pattern were adopted. And then 3D interactive design based on 3D Widgets was given in detail, the implementation view was shown. Finally, some implementation interface and five kinds of widgets design were displayed.

Opportunistic Tangible User Interfaces for Augmented Reality

Opportunistic Controls are a class of user interaction techniques that we have developed for augmented reality (AR) applications to support gesturing on, and receiving feedback from, otherwise unused affordances already present in the domain environment. By leveraging characteristics of these affordances to provide passive haptics that ease gesture input, Opportunistic Controls simplify gesture recognition, and provide tangible feedback to the user. In this approach, 3D widgets are tightly coupled with affordances to provide visual feedback and hints about the functionality of the control. For example, a set of buttons can be mapped to existing tactile features on domain objects. We describe examples of Opportunistic Controls that we have designed and implemented using optical marker tracking, combined with appearance-based gesture recognition. We present the results of two user studies. In the first, participants performed a simulated maintenance inspection of an aircraft engine using a set of virtual buttons implemented both as Opportunistic Controls and using simpler passive haptics. Opportunistic Controls allowed participants to complete their tasks significantly faster and were preferred over the baseline technique. In the second, participants proposed and demonstrated user interfaces incorporating Opportunistic Controls for two domains, allowing us to gain additional insights into how user interfaces featuring Opportunistic Controls might be designed.

A multimodal virtual reality interface for 3D interaction with VTK

The object-oriented visualization Toolkit (VTK) is widely used for scientific visualization. VTK is a visualization library that provides a large number of functions for presenting three-dimensional data. Interaction with the visualized data is controlled with two-dimensional input devices, such as mouse and keyboard. Support for real three-dimensional and multimodal input is non-existent. This paper describes VR-VTK: a multimodal interface to VTK on a virtual environment. Six degree of freedom input devices are used for spatial 3D interaction. They control the 3D widgets that are used to interact with the visualized data. Head tracking is used for camera control. Pedals are used for clutching. Speech input is used for application commands and system control. To address several problems specific for spatial 3D interaction, a number of additional features, such as more complex interaction methods and enhanced depth perception, are discussed. Furthermore, the need for multimodal input to support interaction with the visualization is shown. Two existing VTK applications are ported using VR-VTK to run in a desktop virtual reality system. Informal user experiences are presented.

Precisely Exploring Medical Models and Volumes in Collaborative Virtual Reality

We describe a virtual-reality widget library and two medical applications built on the widget library. These two applications, education using surface models and radiological volume visualization, make use of collaborative interaction techniques. These techniques support a high degree of precision with respect to manipulation of data and data parameters. The 3D widgets instantiated in these applications are synchronized between clients in order to facilitate the high degree of interactivity necessary for productive investigation of shared medical models and volume data. We discuss challenges that face the investigator in an immersive 3D environment as opposed to that of a 2D desktop environment. We describe how these differences have led us to criteria for development of the shared 3D Virtual Reality (VR) graphical user interfaces (GUIs) used in the biomedical applications presented. We review our educational validations already conducted for the surface model exploration application and preview our future work toward a single advanced biomedical collaboration environment.

Frontiers in User‐Computer Interaction

In this age of (near‐)adequate computing power, the power and usability of the user interface is as key to an application’s success as its functionality. Most of the code in modern desktop productivity applications resides in the user interface. But despite its centrality, the user interface field is currently in a rut: the WIMP (Windows, Icons, Menus, Point‐and‐Click GUI based on keyboard and mouse) has evolved little since it was pioneered by Xerox PARC in the early ’70s. Computer and display form factors will change dramatically in the near future and new kinds of interaction devices will soon become available. Desktop environments will be enriched not only with PDAs such as the Newton and Palm Pilot, but also with wearable computers and large‐screen displays produced by new projection technology, including office‐based immersive virtual reality environments. On the input side, we will finally have speech‐recognition and force‐feedback devices. Thus we can look forward to user interfaces that are dramatically more powerful and better matched to human sensory capabilities than those dependent solely on keyboard and mouse. 3D interaction widgets controlled by mice or other interaction devices with three or more degrees of freedom are a natural evolution from their two‐dimensional WIMP counterparts and can decrease the cognitive distance between widget and task for many tasks that are intrinsically 3D, such as scientific visualization and MCAD. More radical post‐WIMP UIs are needed for immersive virtual reality where keyboard and mouse are absent. Immersive VR provides good driving applications for developing post‐WIMP UIs based on multimodal interaction that involve more of our senses by combining the use of gesture, speech, and haptics.

Visualization of Large Nested Graphs in 3D: Navigation and Interaction

Most systems for visualizing large information structures use 2D graphics to view networks of nodes and arcs that represent data. To understand large structures it is often necessary to show both small-scale and large-scale structures. This has been called the problem of focus and context. Distortion, rapid zooming, elision and multiple windows are all techniques that have been developed to provide both focus and context within single representations. We review these techniques and argue that 3D visualization has a number of advantages. A system called NestedVision3D (NV3D) will be presented that has been developed to investigate the use of 3D visualization for understanding the structure of large computer programs.NV3D is a system for visualizing large nested graphs using interactive 3D graphics. It has been tested with graphs containing more than 35,000 nodes and 100,000 relationships. We describe NV3D and its design philosophy. Basic navigation is facilitated by a set of 3D widgets, rapid scaling and interactive elision. More experimental features include animations called snakes, which are used to trace dynamic software behavior.

Object-oriented 3D Modelling, Animation and Interaction

We present an object-oriented 3D graphics and animation framework which provides a new methodology for the symmetric modelling of geometry and behaviour. The toolkit separates the specification of geometry and behaviour by two types of directed acyclic graphs, the geometry graph and the behaviour graph, which are linked together through constraint relations. All geometry objects and behaviour objects are represented as DAG nodes. The geometry graph provides a renderer-independent hierarchical description of 3D scenes and rendering processes. The behaviour graph specifies time- and event-dependent constraints applied to graphics objects. Behaviour graphs simplify the specification of complex animations and 3D interactions by providing nodes for the management of the time and event flow (e.g. durations, time layouts, time repeaters, actions). Nodes contain, manipulate and share instances of constrainable graphical abstract data types. Geometry nodes and behaviour nodes are used to configure high-level 3D widgets, i.e. high-level building blocks for constructing 3D applications. The fine-grained object structure of the system leads to an extensible reusable framework which can be implemented efficiently. © 1997 by John Wiley & Sons, Ltd.

A prototypical 3D graphical visualizer for object-oriented systems

This paper describes a framework for visualizing object-oriented systems within a 3D interactive environment. The 3D visualizer represents the structure of a program as Cylinder Net that simultaneously specifies two relationships between objects within 3D virtual space. Additionally, it represents additional relationships on demand when objects are moved into local focus. The 3D visualizer is implemented using a 3D graphics toolkit, TOAST, that implements 3D widgets and 3D graphics to ease the programming task for 3D visualization.

Magicsphere: an insight tool for 3D data visualization

How to render very complex datasets, and yet maintain interactive response times, is a hot topic in computer graphics. The MagicSphere idea originated as a solution to this problem, but its potential goes much further than this original scope. In fact, it has been designed as a very generical 3D widget: it defines a spherical volume of interest in the dataset modeling space. Then, several filters can be associated with the Magicsphere, which apply different visualization modalities to the data contained in the volume of interest. The visualization of multi-resolution datasets is selected here as a case study and an ad hoc filter has been designed, the MultiRes filter. Some results of a prototipal implementation are presented and discussed.

Using deformations to explore 3D widget design

Using deformations to explore 3D widget design