Photorealistic Environment Research Articles

RD-SLAM: Real-Time Dense SLAM Using Gaussian Splatting

Simultaneous localization and mapping (SLAM) is fundamental for intelligent mobile units to perform diverse tasks. Recent work has shown that integrating neural rendering and SLAM showed promising results in photorealistic environment reconstruction. However, existing methods estimate pose by minimizing the error between rendered and input images, which is time-consuming and cannot be run in real-time, deviating from the original intention of SLAM. In this paper, we propose a dense RGB-D SLAM based on 3D Gaussian splatting (3DGS) while employing generalized iterative closest point (G-ICP) for pose estimation. We actively utilize 3D point cloud information to improve the tracking accuracy and operating speed of the system. At the same time, we propose a dual keyframe selection strategy and its corresponding densification method, which can effectively reconstruct new observation scenes and improve the quality of previously constructed maps. In addition, we introduce regularization loss to address the scale explosion of the 3D Gaussians and over-elongate in the camera viewing direction. Experiments on the Replica, TUM-RGBD, and ScanNet datasets show that our method achieves state-of-the-art tracking accuracy and runtime while being competitive in rendering quality.

Discovering Intrinsic Subgoals for Vision-and-Language Navigation via Hierarchical Reinforcement Learning.

Vision-and-language navigation requires an agent to navigate in a photo-realistic environment by following natural language instructions. Mainstream methods employ imitation learning (IL) to let the agent imitate the behavior of the teacher. The trained model will overfit the teacher's biased behavior, resulting in poor model generalization. Recently, researchers have sought to combine IL and reinforcement learning (RL) to overcome overfitting and enhance model generalization. However, these methods still face the problem of expensive trajectory annotation. We propose a hierarchical RL-based method-discovering intrinsic subgoals via hierarchical (DISH) RL-which overcomes the generalization limitations of current methods and gets rid of expensive label annotations. First, the high-level agent (manager) decomposes the complex navigation problem into simple intrinsic subgoals. Then, the low-level agent (worker) uses an intrinsic subgoal-driven attention mechanism for action prediction in a smaller state space. We place no constraints on the semantics that subgoals may convey, allowing the agent to autonomously learn intrinsic, more generalizable subgoals from navigation tasks. Furthermore, we design a novel history-aware discriminator (HAD) for the worker. The discriminator incorporates historical information into subgoal discrimination and provides the worker with additional intrinsic rewards to alleviate the reward sparsity. Without labeled actions, our method provides supervision for the worker in the form of self-supervision by generating subgoals from the manager. The final results of multiple comparison experiments on the Room-to-Room (R2R) dataset show that our DISH can significantly outperform the baseline in accuracy and efficiency.

Self-Organizing Memory Based on Adaptive Resonance Theory for Vision and Language Navigation

Vision and Language Navigation (VLN) is a task in which an agent needs to understand natural language instructions to reach the target location in a real-scene environment. To improve the model ability of long-horizon planning, emerging research focuses on extending the models with different types of memory structures, mainly including topological maps or a hidden state vector. However, the fixed-length hidden state vector is often insufficient to capture long-term temporal context. In comparison, topological maps have been shown to be beneficial for many robotic navigation tasks. Therefore, we focus on building a feasible and effective topological map representation and using it to improve the navigation performance and the generalization across seen and unseen environments. This paper presents a S elf-organizing Memory based on Adaptive Resonance Theory (SMART) module for incremental topological mapping and a framework for utilizing the SMART module to guide navigation. Based on fusion adaptive resonance theory networks, the SMART module can extract salient scenes from historical observations and build a topological map of the environmental layout. It provides a compact spatial representation and supports the discovery of novel shortcuts through inferences while being explainable in terms of cognitive science. Furthermore, given a language instruction and on top of the topological map, we propose a vision–language alignment framework for navigational decision-making. Notably, the framework utilizes three off-the-shelf pre-trained models to perform landmark extraction, node–landmark matching, and low-level controlling, without any fine-tuning on human-annotated datasets. We validate our approach using the Habitat simulator on VLN-CE tasks, which provides a photo-realistic environment for the embodied agent in continuous action space. The experimental results demonstrate that our approach achieves comparable performance to the supervised baseline.

Perception Action Aware-Based Autonomous Drone Race in a Photorealistic Environment

The development of autonomous, fast, agile small Unmanned Aerial Vehicles (UAVs) brings up fundamental challenges in dynamic environments with fast and agile maneuvers, unreliable state estimation, imperfect sensing, coupling action, and perception in real-time under severe resource constraints. However, autonomous drone racing is a challenging research problem at the intersection of computer vision, planning, state estimation, and control. To bridge this, we propose an approach in the context of autonomous, perception-action aware vision-based drone racing in a photorealistic environment. Our approach integrates a deep convolutional neural network (CNN) with state-of-the-art path planning, state estimation, and control algorithms. The developed deep learning method is based on computer vision approaches to detecting the gates and estimating the flyable area. The planner and controller then use this information to generate a short, minimum-snap trajectory segment and send corresponding motor commands to reach the desired goal. A thorough evaluation of our proposed methodology has been carried out using the Gazebo and FlightGoggles (photorealistic sensor) environments. Extensive experiments demonstrate that the proposed approach outperforms state-of-the-art methods and flies the drone more consistently than many human pilots. Moreover, we demonstrated that our proposed system successfully guided the drone through tight race courses, reaching speeds up to 7m/s of the 2019 AlphaPilot Challenge.

The Effect of Material Properties on the Perceived Shape of Three-Dimensional Objects.

Perceiving the shape of three-dimensional objects is essential for interacting with them in daily life. If objects are constructed from different materials, can the human visual system accurately estimate their three-dimensional shape? We varied the thickness, motion, opacity, and specularity of globally convex objects rendered in a photorealistic environment. These objects were presented under either dynamic or static viewing condition. Observers rated the overall convexity of these objects along the depth axis. Our results show that observers perceived solid transparent objects as flatter than the same objects rendered with opaque reflectance properties. Regional variation in local root-mean-square image contrast was shown to provide information that is predictive of perceived surface convexity.

Vision-based dense displacement and strain estimation of miter gates with the performance evaluation using physics-based graphics models

This paper investigates the framework of vision-based dense displacement and strain measurement of miter gates with the approach for the quantitative evaluation of the expected performance. The proposed framework consists of the following steps: (i) Estimation of 3D displacement and strain from images before and after deformation (water-fill event), (ii) evaluation of the expected performance of the measurement, and (iii) selection of measurement setting with the highest expected accuracy. The framework first estimates the full-field optical flow between the images before and after water-fill event, and project the flow to the finite element (FE) model to estimate the 3D displacement and strain. Then, the expected displacement/strain estimation accuracy is evaluated at each node/element of the FE model. Finally, methods and measurement settings with the highest expected accuracy are selected to achieve the best results from the field measurement. A physics-based graphics model (PBGM) of miter gates of the Greenup Lock and Dam with the updated texturing step is used to simulate the vision-based measurements in a photo-realistic environment and evaluate the expected performance of different measurement plans (camera properties, camera placement, post-processing algorithms). The framework investigated in this paper can be used to analyze and optimize the performance of the measurement with different camera placement and post-processing steps prior to the field test.

Next generation soundscape design using virtual reality technologies

Sound quality has been demanded by the community as part of the smart city initiatives to enjoy a real liveable environment. While a lot of governments pay attention to environmental issues such as air quality and waste, those who care about the sonic environment still rely primarily quantitative levels for either compliance or improvement. Sound quality is most likely ignored. There is at present the international standard ISO 12913-1:2014 providing the definition and the conceptual framework of soundscape. Despite the situation, sound quality is a subjective matter after all and relies very much on human perceptions, as well as the contextual environment. Sound walk, questionnaire, and lab test are common tools used in soundscape studies. These tools allow a good understanding of the perception of the prevailing sonic environment. To figure the sound quality of different design options, though, we can immerse the participants in a virtual, but photorealistic environment, so that they can perceive, compare, and give feedback as if they were in the real environment. A mobile VR application has been developed for iterating soundscape design and fine-tuning. This paper presents this application with case studies.

Belief revision and way-finding

Belief revision is required when veridical information surfaces that contradicts what was previously thought to be the case. In way-finding, belief revision frequently occurs, for example, when the travelled route has led one astray, instead of to one's chosen destination. In past cognitive research, the topics of belief revision and way-finding have been treated in isolation. Here, we introduce an approach for linking the two fields and assess belief revision as it occurs in the process of way-finding. We report the results of two experiments that put participants in (virtual) situations where elements of a previously learned route description do not match the actual environment (thereby requiring the revision of a previously held belief). Experiment 1 puts participants in a highly artificial virtual environment where the landmarks to be used in navigation have a low degree of semantic salience (houses of various color). Experiment 2 puts subjects in a photorealistic environment where the objects to be used in navigation are well-known landmarks (such as the Eiffel Tower) and thus have a high degree of semantic salience. In both experiments, participants are confronted with T-junctions, where a landmark that was expected to indicate the correct route is discovered to be in an unexpected location. The results of the experiments show that a participant's choice of route, in such cases, is affected by differences in the structure of the relevant initial instruction. More precisely, the route chosen by participants is affected by whether the relevant landmark was described as being on the same side of the path as they were instructed to turn (congruent case) or as located on the opposite side of the path as they were instructed to turn (incongruent case).

Designing (for) experiences in photorealistic VR environments

This paper investigates the role of aesthetics in the design of “intended” experiences in photorealistic virtual reality (VR) environments. It is motivated by the very notion that the aesthetic potential of photorealistic VR content is, and continues to be, underestimated whilst the emphasis on the development of newer and more efficient visualisation technologies to create new and exciting VR experiences increases. Challenging this, the paper looks beyond the technological (and the more traditional human computer interaction approaches that have primarily focused on the performance and efficiency issues of the technology) in order to explore more human values and the experiential side of VR. It focuses on the design of an “engaged interaction” and in doing so, implements a comparative study to explore how the strategic patterning of the aesthetic elements (particularly colour) within a photorealistic VR environment can allow for the design of a certain experience. In conclusion, the paper demonstrates that aesthetics and the “engaged interaction” can play an important role in getting to the heart of the photorealistic VR “user” experience. It highlights how we might design for (i.e. suggest, coax and guide) an “intended” VR experience.

User Friendly 3-D Interior Image Generation System Using Photograph Image.

A graphic model generation system for house interiors has been developed. The system generates an interior image which fits a user's idea. The system includes an interior picture database which can be accessed interactively using key words to select a desired picture. The system generates a graphic model by mapping part images included in the selected picture on a specified room structure. The dimensions and view points of the picture in the database and the specified room are different. Therefore, the graphic model is three-dimensional. To construct a three-dimensional parts image database, a picture image is recognized by a monocular vision algorithm, and normalized images, sizes, and locations in a room space of the picture are calculated. Using these data, part images can be mapped on a specified room structure. The system was utilized in a home electronics presentation system to generate a photorealistic environment model. This environment model helps in appearence design evaluation and setting of product specifications to fill customer requirements.