3D User Research Articles

Webcam-based gaze estimation for computer screen interaction.

This paper presents a novel webcam-based approach for gaze estimation on computer screens. Utilizing appearance based gaze estimation models, the system provides a method for mapping the gaze vector from the user's perspective onto the computer screen. Notably, it determines the user's 3D position in front of the screen, using only a 2D webcam without the need for additional markers or equipment. The study presents a comprehensive comparative analysis, assessing the performance of the proposed method against established eye tracking solutions. This includes a direct comparison with the purpose-built Tobii Eye Tracker 5, a high-end hardware solution, and the webcam-based GazeRecorder software. In experiments replicating head movements, especially those imitating yaw rotations, the study brings to light the inherent difficulties associated with tracking such motions using 2D webcams. This research introduces a solution by integrating Structure from Motion (SfM) into the Convolutional Neural Network (CNN) model. The study's accomplishments include showcasing the potential for accurate screen gaze tracking with a simple webcam, presenting a novel approach for physical distance computation, and proposing compensation for head movements, laying the groundwork for advancements in real-world gaze estimation scenarios.

VirtualCube: An Immersive 3D Video Communication System.

The VirtualCube system is a 3D video conference system that attempts to overcome some limitations of conventional technologies. The key ingredient is VirtualCube, an abstract representation of a real-world cubicle instrumented with RGBD cameras for capturing the user's 3D geometry and texture. We design VirtualCube so that the task of data capturing is standardized and significantly simplified, and everything can be built using off-the-shelf hardware. We use VirtualCubes as the basic building blocks of a virtual conferencing environment, and we provide each VirtualCube user with a surrounding display showing life-size videos of remote participants. To achieve real-time rendering of remote participants, we develop the V-Cube View algorithm, which uses multi-view stereo for more accurate depth estimation and Lumi-Net rendering for better rendering quality. The VirtualCube system correctly preserves the mutual eye gaze between participants, allowing them to establish eye contact and be aware of who is visually paying attention to them. The system also allows a participant to have side discussions with remote participants as if they were in the same room. Finally, the system sheds lights on how to support the shared space of work items (e.g., documents and applications) and track participants' visual attention to work items.

An integrated mixed reality system for safety-aware human-robot collaboration using deep learning and digital twin generation

For human-robot collaboration (HRC), one of the most practical methods to ensure human safety with a vision-based system is establishing a minimum safe distance. This study proposes a novel integrated mixed reality (MR) system for safety-aware HRC using deep learning and digital twin generation. The proposed approach can accurately measure the minimum safe distance in real-time and provide MR-based task assistance to the human operator. The approach integrates MR with safety-related monitoring by tracking the shared workplace and providing user-centric visualization through smart MR glasses for safe and effective HRC. Two RGB-D sensors are used to reconstruct and track the working environment. One sensor scans one area of the physical environment through 3D point cloud data. The other also scans another area of the environment and tracks the user's 3D skeletal information. In addition, the two partially scanned environments are registered together by applying a fast global registration method to two sets of the 3D point cloud. Furthermore, deep learning-based instance segmentation is applied to the target object's 3D point cloud to increase the registration between the real robot and its virtual robot, the digital twin of the real robot. While only 3D point cloud data are widely used in previous studies, this study proposes a simple yet effective 3D offset-based safety distance calculation method based on the robot's digital twin and the human skeleton. The 3D offset-based method allows for real-time applicability without sacrificing the accuracy of safety distance calculation for HRI. In addition, two comparative evaluations were conducted to confirm the originality and advantage of the proposed MR-based HRC.

USER REQUIREMENTS GATHERING FOR A NATIONAL 3D MAPPING PRODUCT IN THE UNITED KINGDOM

Abstract. One of the main barriers to the uptake of 3D GI is the lack of understanding of what the user requirements are. From the data acquisition and creation perspective – in particular, that of a National Mapping and Cadastral Agency who may need to prepare datasets with national coverage – this is an issue as each new 3D feature type and element within a feature added (such as doors, windows, chimneys, street lights) requires additional processing and cost to create. This paper reports the results of a user requirements gathering exercise for a national 3D mapping product in the United Kingdom. The study focuses on the user perception of ‘usefulness’ of different 3D geometry and semantic features. A web-based questionnaire with Likert-type items was selected as the primary data collection method and was conducted in May 2017. A total of 121 completed responses were from the UK. Descriptive analysis showed that ‘Air quality engineering’, ‘Infrastructure & transport’ and ‘Environmental services’ presented the most positive outlook on the usefulness of 3D. Correlation analysis showed that potential 3D product groups that could be formed in a multi-product approach. Cluster analysis showed that appetite for 3D information not only varies between sectors, but also within sectors between different practitioners. The results from exploratory factor analysis showed that users were more interested in additional information on non-building features rather than additional detail to building geometry. Further continued work is required to incorporate both non-GIS users and 3D users outside the GIS domain.

EyePACT

The parallax effect describes the displacement between the perceived and detected touch locations on a touch-enabled surface. Parallax is a key usability challenge for interactive displays, particularly for those that require thick layers of glass between the screen and the touch surface to protect them from vandalism. To address this challenge, we present EyePACT, a method that compensates for input error caused by parallax on public displays. Our method uses a display-mounted depth camera to detect the user's 3D eye position in front of the display and the detected touch location to predict the perceived touch location on the surface. We evaluate our method in two user studies in terms of parallax correction performance as well as multi-user support. Our evaluations demonstrate that EyePACT (1) significantly improves accuracy even with varying gap distances between the touch surface and the display, (2) adapts to different levels of parallax by resulting in significantly larger corrections with larger gap distances, and (3) maintains a significantly large distance between two users' fingers when interacting with the same object. These findings are promising for the development of future parallax-free interactive displays.

Application of Integration of HBIM and VR Technology to 3D Immersive Digital Management—Take Han Type Traditional Architecture as an Example

Abstract. HBIM technology makes great contributions to 3D digital preservation and management of the existing traditional architectures, and VR technology has also been gradually emphasized by 3D users in recent years, especially 3D immersive situation makes users more likely to experience the real space field. Taking Han type traditional architecture with relatively complex geometrical structure as an example, this research carries out digital preservation through HBIM technology and tries to switch to VR platform to allow users to enter 3D immersive scene for management and display. It is shown in the research results that the application of integration of HBIM and VR technology to Han type traditional architecture needs to consider 3D digital model of the architecture, and the number of polygon shall be controlled below about 2 million, which can make the operation in VR environment more smooth; the integration of two technologies can achieve the purpose of 3D immersive digital management, which can provide the humanized application close to the real experience for the display of subsequent management of ancient relics and architectural aesthetics.

User Acceptance of Technologies in Their Infancy

This paper adjusts UTAUT2 to study user acceptance of an early stage technology by using the example of 3D printing for home users. The authors' findings show that UTAUT2 is capable of measuring user acceptance and that social influence and effort expectancy are good predictors for ‘intention to use' for early stage technologies. Nevertheless, there seem to be significant differences in the antecedents depending on the involvement levels and the DIY mentality of the user. Moreover, this research confirms and extends the applicability of UTAUT2 to early stage technologies while improving the authors' understanding of prospective 3D printing home users.

SAGE: Semantic Annotation of Georeferenced Environments

The emergence of portable 3D mapping systems are revolutionizing the way we generate digital 3D models of environments. These systems are human-centric and require the user to hold or carry the device while continuously walking and mapping an environment. In this paper, we adapt this unique coexistence of man and machines to propose SAGE (Semantic Annotation of Georeferenced Environments). SAGE consists of a portable 3D mobile mapping system and a smartphone that enables the user to assign semantic content to georeferenced 3D point clouds while scanning a scene. The proposed system contains several components including touchless speech acquisition, background noise adaptation, real time audio and vibrotactile feedback, automatic speech recognition, distributed clock synchronization, 3D annotation localization, user interaction, and interactive visualization. The most crucial advantage of SAGE technology is that it can be used to infer dynamic activities within an environment. Such activities are difficult to be identified with existing post-processing semantic annotation techniques. The capability of SAGE leads to many promising applications such as intelligent scene classification, place recognition and navigational aid tasks. We conduct several experiments to demonstrate the effectiveness of the proposed system.

Property Changes of Woven Blouse Fabrics by bonding fusible interlinings for a 3D Virtual Try-on System

This study aims to analyze the changes in the mechanical properties of woven fabrics(cotton, linen, wool, silk, and polyester) by bonding fusible interlinings with varying deniers(10D, 20D, and 30D) for a 3D virtual try-on system(one that a user to try garments through screens using Avatar) developed over the last decade. We experimented with four mechanical properties and thicknesses of twenty-three specimens of interlining bonded fabrics including face fabrics and interlinings by using the KES-FB-AUTO system. The results showed that the tensile property increased(LT and RT increased and WT decreased) as the denier of the interlining increased; however, the change was slight. In contrast, the bending and shear properties increased significantly as the denier of the interlining increased on both the warp and the weft. This showed evidence that the interlining gives the fabrics size stability. The compression property was slight changed as the tensile property varies depending on the fibers and the denier of interlining. As expected, the thickness increased by bonding the interlining as the denier of interlining increased. From these results, we conclude that 3D users need to reflect these changes of woven fabrics by bonding interlinings when they try screen fittings to accurately express the fabric reality of manufactured garment.

Study and Application on 3D Printing Cloud Service Platform

Development of 3D printing technology rapidly and widely used in machinery manufacturing. 3D printing business is much, but the efficient equipment is deficiency. Methods of 3D models collecting, processing and online displaying were studied. A 3D printing service platform was constructed based on Cloud computing technology. The service program was developed by ASP. The background of the platform was managed by MySQL database. CAD files were displayed on a HTML5 browser by WebGL. A 3D printer owner released the 3D printer information, users searched printers and uploaded models data. The models were manufactured by remote online printers. The platform provides excellent products manufacturing environment for 3D printer owners and users. Products and printers are full information sharing. The economic benefit is rich.

A Model for Managing 3D Printing Services in Academic Libraries.

The appearance of 3D printers in university libraries opens many opportunities for advancing outreach, teaching, and research programs. The University of Alabama (UA) Libraries recently adopted 3D printing technology and maintains an open access 3D Printing Studio. The Studio consists of a 3D printer, multiple 3D design workstations, and other supporting equipment. Training of new Studio users (students, faculty, and staff) is a two-step process: an initial workshop followed by an individual training session. After the individual meeting(s), users may access equipment in the Studio independently. The 3D Printing Studio service is popular across campus. During early implementation, 50 users have attended training workshops. 3D users have experimented in the Studio with projects for advanced coursework and independent research. It is expected that the number of trained users will double in the near future. An evaluation of our management policies suggests that providing an open access environment and permitting users to experiment independently in the 3D Printing Studio contributes greatly to the success of the service. We also present ideas for future improvements to the 3D Studio service. [ABSTRACT FROM AUTHOR]

Managing 3D Multi User Learning Environments - A Case Study on Training Disaster Management

3D Multi User Virtual Environments (3D MUVE), or commonly known as 3D virtual worlds, have shown proven success in enhancing the present teaching and learner support methodologies. Integrated educational environments that are formed using 3D MUVE can be referred as 3D Multi User Learning Environments (3D MULE). The intrinsic rich and dynamic features of 3D virtual worlds can sufficiently increase the student engagement with the learning tasks in 3D MULE. However, with the facilitation for diverse learning activities, 3D MULE can introduce a new set of challenges for the teachers and students; therefore, suitable management strategies for 3D MULE can be essential for success. In this research, we have proposed a framework for managing 3D MULE using policy considerations and a guidance tool to facilitate policy implementations. This paper presents the evaluation of the proposed strategies for making policy considerations to manage 3D MULE and the developed user guidance tool. To increase the accuracy while evaluating for real educational uses, we selected a teaching and training environment in use as the case study. This case study, a 3D MULE supported learning aid for humanitarian disaster management, provided valuable and supportive feedback to validate the completed work while shaping the orientation of the future research on facilitating 3D MULE management.

깊이맵 센서를 이용한 3D캐릭터 가상공간 내비게이션 동작 합성 및 제어 방법

키넥트의 성공적인 등장 이후 이 센서를 이용하여 사용자의 아바타에 해당하는 3차원 캐릭터의 움직임을 제어하는 많은 인터액티브 콘텐츠가 제작되었다. 하지만, 키넥트의 특성 상 사용자는 키넥트를 정면으로 바라보아야 하며, 모션 또한 제자리에서 수행할 수 있는 동작 정도만으로 국한되었다. 이 단점은 게임에서 가장 중요한 요구기능 중 하나인 가상공간 내비게이션을 수행하지 못하게 하는 근본적인 이유가 되었다. 본 논문은 이와 같은 단점을 해결하기 위한 새로운 방법을 제안한다. 두 단계로 이루어진 본 방법은 첫 번째 단계로서 사용자의 내비게이션 의도를 파악하기 위해 제자리 걷기 동작 제스처인식을 수행한다. 내비게이션 의도가 파악되면, 다음 단계에 현재 제자리 걷기동작을 상체와 하체 모션으로 자동으로 분리한 후, 미리 입력 받은 하체모션캡처 데이터를 현재 캐릭터 속도를 반영하여 수정한 뒤 분리된 원래 하체모션과 자연스럽게 교체한다. 본 논문에서 제안된 알고리즘을 이용하면, 키넥트 센서를 통해 사용자의 상체 모션을 그대로 반영함과 동시에 모션캡처 데이터를 이용하여 하체 동작을 실제 걷는 동작으로 바꾸어주기 때문에 사용자가 조정하는 3차원 캐릭터는 가상공간을 자연스럽게 내비게이션할 수 있다. After successful advent of Microsoft's Kinect, many interactive contents that control user's 3D avatar motions in realtime have been created. However, due to the Kinect's intrinsic IR projection problem, users are restricted to face the sensor directly forward and to perform all motions in a standing-still position. These constraints are main reasons that make it almost impossible for the 3D character to navigate the virtual environment, which is one of the most required functionalities in games. This paper proposes a new method that makes 3D character navigate the virtual environment with highly realistic motions. First, in order to find out the user's intention of navigating the virtual environment, the method recognizes walking-in-place motion. Second, the algorithm applies the motion splicing technique which segments the upper and the lower motions of character automatically and then switches the lower motion with pre-processed motion capture data naturally. Since the proposed algorithm can synthesize realistic lower-body walking motion while using motion capture data as well as capturing upper body motion on-line puppetry manner, it allows the 3D character to navigate the virtual environment realistically.

Application of 3D Stereo Visualization System in Virtual Architectural Design

The application of virtual reality in architectural design has been the focus of attention. But the true 3D sense of immersion is still a dream for the most designers and clients when they watch the computer screen. A new computer stereo vision imaging system was constructed to help us obtain 3D visual of architectures and the implementation process became easier. And the application of advanced three-dimensional vision technology offered a more scientific, more interesting means of display and human-computer interaction. By using 3D stereo vision imaging system, all designers and users can obtain 3D visual observation of virtual buildings and virtual environments with a good sense of immersion at any time and any place.

The Development of a Virtual Marine Museum for Educational Applications

The objective of this article is to investigate the computer animation and virtual reality technologies for developing a virtual marine museum. The museum consists of three exhibition areas. The first area displays fishes in freshwater, including creeks, rivers, and dams in Taiwan. The second area exhibits marine ecology and creatures of different species in the ocean. The third area is an undersea transparent tunnel where one can see sharks, tuna, and lovely butterfly fish swimming around beautiful coral reefs. Since the museum is developed by computer animation and virtual reality technology, it provides 3D visual effects and interactive user interface. People can breed their favorite fish in the virtual aquarium, and make observations through networks for educational and recreational purposes. A qualitative study was conducted and the results showed that the virtual marine museum could raise students' interest and learning motivation. Therefore, it can be used as an assistant tool in science education for learning marine ecology and promoting ecological protection of oceanic environments as well.

Development and control of a versatile nanohandling robot cell

Current research work on the development of a nanohandling robot cell in a scanning electron microscope (SEM) is presented. An experimental setup is shown, in which two microrobots cooperate in the vacuum chamber of an SEM. One microrobot is used as manipulator of the tool and a second one is used for manipulating the object. Priority objective in future is the systematic prototyping of CNT-based nanodevices. Special algorithms for real-time processing of noisy SEM images have been implemented and tested. In addition, a 3D graphical user interface has been developed and delivers depth information which is often crucial for teleoperation of nanoobjects due to the lack of depth information in original SEM images. In order to enable automated nanohandling, a new control strategy is proposed. One key challenge for a successful automation is continuous pose estimation of tool and object which can be solved by applying the developed 2D and 3D tracking algorithms. The functionality of the robot cell is exemplified by handling and mechanical characterization of carbon nanotubes (CNTs).

Study of Three-Dimensional Stress Distribution and Damage Characterization of Bolt Composite Joint

A 3D finite element model of bolt composite joint has been established to determine the stress distribution on the contact surface. The effects of clamping torque and friction on the contact stress and interlaminar normal stress are considered. From the analysis results, contact stress is bared mainly by the 0° layer. The distribution and magnitude of contact stress are conducted by friction. The effect of clamping torque on interlaminar normal stress is very strong. A 3D damage user subroutine is added to the FEM to simulate the damage of joint. By the means of damage simulation, the initiation and progression direction of three types damage are predicted. Matrix cracking and fiber-matrix shear occur at first, and fiber buckling is founded subsequently. The matrix cracking and fiber-matrix debonding initiate at circumferential angle 45°and 135°, and fiber buckling initiate at the 0° layer on the bearing plane. The friction and bolt clamping torque can restrain damage initiation and development.

Jabiru: Harnessing Java 3D Behaviors for Device and Display Portability

The aim of this paper is to use the Java 3D ConfiguredUniverse utilities to create a set of behaviors that we can integrate into existing Java 3D programs, as long as these programs use the ConfiguredUniverse. We call this behavior package Jabiru (Java 3D Application Behavior Immersive Virtual Reality Utilities). The Jabiru set of behaviors, accessible from a master menu, facilitates moving Java 3D applications originally created for desktop environments to immersive VR environments and vice versa. Jabiru also provides a six-degrees-of-freedom (6DOF) device emulator for a conventional mouse, meant to facilitate offsite testing of immersive VR behaviors on a desktop. The main focus of our work with a CAVE is in relation to bioinformatics, which has embraced Java and Java 3D as one of the choice programming environments. By creating this package, we help bring 3D graphics to both the bioinformatics community and the casual Java 3D developer or user. We discuss the design and implementation of Jabiru.

Motion prediction for caching and prefetching in mouse-driven DVE navigation

A distributed virtual environment (DVE) allows geographically separated users to participate in a shared virtual environment via connected networks. However, when the users are connected by the Internet, bandwidth limitation and network latency may seriously affect the performance and the interactivity of the system. This explains why there are very few DVE applications for the Internet. To address these shortcomings, caching and prefetching techniques are usually employed. Unfortunately, the effectiveness of these techniques depends largely on the accuracy of the prediction method used. Although there are a few methods proposed for predicting 3D motion, most of them are primarily designed for predicting the motion of specific objects by assuming certain object motion behaviors. We notice that in desktop DVE applications, such as virtual walkthrough and network gaming, the 2D mouse is still the most popular device used for navigation input. Through studying the motion behavior of a mouse during 3D navigation, we have developed a hybrid motion model for predicting the mouse motion during such navigation---a linear model for prediction at low-velocity motion and an elliptic model for prediction at high-velocity motion. The predicted mouse motion velocity is then mapped to the 3D environment for predicting the user's 3D motion. We describe how this prediction method can be integrated into the caching and prefetching mechanisms of our DVE prototype. We also demonstrate the effectiveness of the method and the resulting caching and prefetching mechanisms through extensive experiments.

Session 5: 3D image visualization and user interfaces

Session 5: 3D image visualization and user interfaces