Real-time 3D Environment Research Articles

EchoSee: An Assistive Mobile Application for Real-Time 3D Environment Reconstruction and Sonification Supporting Enhanced Navigation for People with Vision Impairments.

Improving the quality of life for people with vision impairments has been an important goal in the research and design of assistive devices for several decades. This paper seeks to further that goal by introducing a novel assistive technology platform that leverages real-time 3D spatial audio to promote safe and efficient navigation for people who are blind or visually impaired (PVI). The presented platform, EchoSee, uses modern 3D scanning technology on a mobile device to construct a live, digital 3D map of a user's environment as they move about their surroundings. Spatialized, virtual audio sources (i.e., virtual speakers) are dynamically placed within the digital 3D scan of the world, providing the navigator with a real-time 3D stereo audio "soundscape." The digital 3D map, and its resultant soundscape, are continuously updated as the user moves about their environment. The generated soundscape is played back through headphones connected to the navigator's device. This paper details (1) the underlying technical components and how they were integrated to produce the mobile application that generates a dynamic soundscape on a consumer mobile device, (2) a methodology for analyzing navigation performance with the application, (3) the design and execution of a user study investigating the effectiveness of the presented system, and (4) a discussion of the results of that study along with a proposed future study and possible improvements. Altogether, this paper presents a novel software platform aimed at assisting individuals with vision impairments to navigate and understand spaces safely, efficiently, and independently and the results of a feasibility study analyzing the viability of the approach.

Digital twin framework for smart greenhouse management using next-gen mobile networks and machine learning

Due to the increase in world population, arable land has been reduced. Consequently, the concept of urban greenhouses is on the rise. Smart greenhouses need to monitor physical parameters for the healthy growth of plants from remote locations. A digital twin is a representation of physical assets in the digital world, and this emerging technology has opened up opportunities for efficient system development for Industry 4.0. The digital twin receives real-time operational data to monitor the asset in the digital domain. It performs real-time processing, data analysis, and machine learning to predict optimized decisions. In the era of next-generation mobile networks, IoT devices can communicate and perform their remote operations in a timely manner. In smart greenhouse technology, the digital twin could be a revolutionary substitute for real-time remote monitoring and process management. However, there has been limited work on digital twin-driven smart greenhouse technology. In this paper, a process management framework is developed that can be interpreted as a machine learning and cloud-based data-driven digital twin for smart greenhouses. The proposed framework consists of three layers: the physical, fog, and cloud layers. The physical greenhouse measurements are monitored using a highly immersive cloud-based, real-time 3D environment. We present an example architecture using commercial cloud and open-source tools to verify the proof of concept. Additionally, different ML techniques are utilized to predict the operational requirements for smart greenhouses.

RobNet: real-time road-object 3D point cloud segmentation based on SqueezeNet and cyclic CRF

In order to realize real-time 3D environment perception of UAVs and autopilot in low-altitude complex road scenes, a neural network model RobNet based on SqueezeNet and cyclic CRF for real-time 3D point cloud segmentation is proposed to segment the road objects in real time. Firstly, the unordered, scattered 3D point cloud data are preprocessed into a standard data format similar to an image by a spherical mapping method. Then, at the macro-level of the model design, the SqueezeNet network with the residual connection optimization is selected as the basic network of the model, and then, the conditional random field (CRF) algorithm which is processed into the cyclic network structure is used to refine the segmentation result. Finally, the construction of the basic network, the cyclic network and the network parameter settings in the model is elaborated at the micro-level. The experimental results show that the RobNet model proposed in this paper can segment the target in the road scene better. The segmentation callback rate of the three types of vehicles, pedestrians and cyclists is increased by 28, 2 and 17%, respectively, compared with the VoxelNet network. The higher callback rate is in line with the safe movement specifications for drones and autonomous driving. At the same time, the proposed model parameters are small, 98.5% smaller than the classic network AlexNet, and are easy to deploy on a platform with limited computing resources. The RobNet model in the Robot Operating System (ROS) framework engineering deployment and implementation experimental data shows that the model meets the real-time and stability requirements of the drone and automatic driving application, engineering code can run in real time at 12 Hz, the standard deviation of each frame’s running time is around 4.5 ms.

形状の特徴や動的な変形を考慮したリアルタイム3DCGにおける輪郭線の誇張表現手法

In two-dimensional (2D) animation, the thickness of object outlines and contour lines is often varied to emphasize or exaggerate the shape of the object. Adding a 2D touch to three-dimensional computer graphics (3DCG) animation using toon-rendering is common, but existing techniques cannot achieve varying line thickness. This paper proposes a method of recreating the effect of varying the outline and contour line thicknesses based on the object shape in real-time 3D environments such as video games. Our method focuses on exaggerating the thickness of lines where the target polygon is curved. Furthermore, by using a backface polygon of the target polygon, we are able to create a continuous outline of the object, thus recreating a closed outline of the object with varying line thickness.

Eight Questions (and Answers) about Machinima

The author argues that, unlike traditional filmmaking or video recording through inexpensive digital videocameras, machinima presents a high barrier of entry, remaining a relatively complex method of creating videos. In fact, producing rendered animations generated from recordings of gameplay requires fairly high-end hardware capable of running real-time graphics intensive 3D or pseudo-3D environments. Nonetheless, machinima has the potential to make a real impact on the political landscape and can be considered a relatively sophisticated form of art, although one has to remember that art is socially constructed and it is not an intrinsic quality of an artifact.

SIEVE: Collaborative Decision-making in an Immersive Online Environment

Over the recent years, spatial data have become more accessible to members of the public, and there is an increased awareness that spatial data are an essential ingredient in the development of governance policies. In parallel, computing devices that can render real-time 3D environments in nearly photo-realistic quality have become a common household item. Integrating spatial data and virtual environments opens the possibility of presenting spatial data in a highly visual way—the same way we experience our everyday world. We have developed a landscape planning tool called SIEVE (Spatial Information and Visualization Environment), which allows users to explore existing spatial data and hypothetical future scenarios in a real-time 3D environment, links to environmental process models outcomes, and also provides a collaborative decision-making environment. SIEVE has an automatic model-building component that allows users to select a region on a 2D map using a web portal and to download automatically generated 3D landscape environments. SIEVE not only models above-ground features but also incorporates underground features. This allows users to see the connection between below- and above-ground processes. Lastly, SIEVE incorporates a multi-user environment that allows users from different locations to gather in the virtual landscapes for exploring and decision-making purposes.

Driver Training Simulator for Backing Up Commercial Vehicles with Trailers

Backing up tractors with trailers is a difficult task since the kinematic behavior of articulated vehicles is complex and hard to control. Especially unskilled drivers are overstrained with the complicated steering process. To learn and practice the steering behavior of articulated vehicles, we developed a 3D driving simulator. The simulator can handle different types of articulated vehicles like semi-trailers, one- and two-axle trailers, or gigaliners. The use of a driving simulator offers many advantages over the use of real vehicles. One of the main advantages is the possibility to learn the steering behavior of all vehicle types. Drivers can be given more and better driving instructions like collision warnings or steering hints. Furthermore, the driver training costs can be reduced. Moreover, mistakes of the student do not lead to real damages and costly repairs. The hardware of the simulator consists of a low cost commercial driving stand with original truck parts, a projection of the windshield and two flat panel monitors for the left and right exterior mirrors. Standard PC hardware is used for controlling the driving stand and for generating the realtime 3D environment. Each aspect of the simulation like realistic vehicle movements or generation of different views, is handled by a specific software module. This flexible system can be easily extended which offers the opportunity for other uses than just driver training. Therefore, we use the simulator for the development and test of driver assistance systems.

A novel biologically inspired neural network solution for robotic 3D sound source sensing

This paper presents a novel real-time robotic binaural sound localization method based on hierarchical fuzzy artificial neural networks and a generic set of head related transfer functions. The robot is a humanoid equipped with the KEMAR artificial head and torso. Inside the ear canals two small microphones play the role of the eardrums in collecting the impinging sound waves. The neural networks are trained using synthesized sound sources placed every 5° from 0° to 255° in azimuth, and every 5° from − 45° to 80° in elevation. To improve generalization, the training data was corrupted with noise. Thanks to fuzzy logic, the method is able to interpolate at its output, locating with high accuracy sound sources at positions which were not used for training, even in presence of strong distortion. In order to achieve high localization accuracy, two different binaural cues are combined, namely the interaural intensity differences and interaural time differences. As opposed to microphone-array methods, the presented technique, uses only two microphones to localize sound sources in a real-time 3D environment.

An implementation of real-time 3D interactive drama

Interactive fiction and adventure video games are narrative genres which provide the player with the option of acting as the main character of the story. However these genres do not fully match the expectations of their authors and readers because the player cannot deeply affect the storyline. This article describes a system integrating highly interactive narrative structures in a real-time 3D environment. Based on a theoretical foundation of narrative and drama, an interactive drama engine (IDE) has been implemented. It comprises an action calculus system, a text-generation system, a behavior engine, an animation engine as well as an innovative adaptive user interface. The IDE is demonstrated with two scenarios.

A Plausible Lightweight Dirt Model for Application in Real-time 3D Environments

Dirt is an important, if unattractive, feature of our real world, the presence of which provides key subliminal cues that enhance understanding of many factors including age, history, economic and social status. In addition, the accumulation of dirt may also aid in the perception of fine detail, on complex surfaces, by highlighting fine features. Because dirt is such an important factor in the manner in which we perceive the real world, its inclusion in 3D virtual environments should significantly increase the sense of presence a user experiences by creating a more plausible impression of the real world environment. Most traditional approaches for incorporating dirt into 3D environments rely on either highly complex, but accurate, simulations, or depend on artistic skill. This research seeks to identify an alternative approach, grounded in human perception, which evaluates simplified approximations of dirt aggregation with the aim of identifying an approximate model that presents a plausible representation of dirt. This paper describes the process by which such an approximation has been identified and evaluated, and presents surprising results as to the level of complexity such a model requires to be viable.

Using wireless technology to develop a virtual reality command and control centre

This paper investigates the applicability of wireless communication systems for use in command and control environments. Human positional data is transmitted over a wireless network. This data is then used to update a highly accurately modelled real-time 3D environment of the surroundings, with avatars positioned at the transmitted points. The data is displayed on a stereoscopic 3D screen enabling novel automatic tracking of human movement and allowing for more rapid and informed tactical decision making. The system has applicability in a variety of C4I environments, including the military and emergency services.