Real-time Viewpoint Research Articles

Dimensionality Reduction for the Real-Time Light-Field View Synthesis of Kernel-Based Models

Several frameworks have been proposed for delivering interactive, panoramic, camera-captured, six-degrees-of-freedom video content. However, it remains unclear which framework will meet all requirements the best. In this work, we focus on a Steered Mixture of Experts (SMoE) for 4D planar light fields, which is a kernel-based representation. For SMoE to be viable in interactive light-field experiences, real-time view synthesis is crucial yet unsolved. This paper presents two key contributions: a mathematical derivation of a view-specific, intrinsically 2D model from the original 4D light field model and a GPU graphics pipeline that synthesizes these viewpoints in real time. Configuring the proposed GPU implementation for high accuracy, a frequency of 180 to 290 Hz at a resolution of 2048×2048 pixels on an NVIDIA RTX 2080Ti is achieved. Compared to NVIDIA’s instant-ngp Neural Radiance Fields (NeRFs) with the default configuration, our light field rendering technique is 42 to 597 times faster. Additionally, allowing near-imperceptible artifacts in the reconstruction process can further increase speed by 40%. A first-order Taylor approximation causes imperfect views with peak signal-to-noise ratio (PSNR) scores between 45 dB and 63 dB compared to the reference implementation. In conclusion, we present an efficient algorithm for synthesizing 2D views at arbitrary viewpoints from 4D planar light-field SMoE models, enabling real-time, interactive, and high-quality light-field rendering within the SMoE framework.

Anomaly Detection of Tire Tiny Text: Mechanism and Method

Due to the variety of tire specifications and poor contrast, it is difficult for ordinary 2D vision-based methods to accurately and autonomously detect anomalies in the tire text embossed on the tire sidewalls. In this paper, a 3D vision-based autonomous scanning mechanism and method is proposed, including three core components: a three-degree-of-freedom (3-DoF) robotic arm, a high-precision turntable and a composite vision probe. Based on the designed vision probe, the mechanism can autonomously scan the tire sidewall through a series of optimal viewpoints to obtain high-quality tire images even in the absence of tire CAD models. To obtain the optimal viewpoints, an information-driven viewpoint planning method is proposed. Combined with the real-time viewpoint adjustment strategy, the viewpoint of the vision probe is dynamically fine-tuned, which can effectively balance the scanning precision and motion cost. In the text anomaly detection phase, a three-stage tire text anomaly detection method based on deep learning and statistics strategy is proposed, which can accurately detect and recognize tire text, identify sub-mold categories, and judge text anomalies. Experimental results show that the proposed mechanism and method perform well in terms of accuracy ( <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$&gt;$</tex-math> </inline-formula> 96%) and efficiency (time per lap for a tire is less than 9 s); it shows that the proposed mechanism and method have certain industrial application value and can be applied to tire quality inspection of on-line production. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —Tire text may be incorrectly embossed during tire manufacturing. Due to the wide variety of tire specifications, poor contrast, and thin text strokes, it is difficult to detect abnormal tire text. In this paper, a 3D vision-based mechanism and method is developed to address this issue. Using a specially designed composite vision probe and scanning method, the mechanism can observe tiny text information on tires of different specifications from multiple angles and directions. By using our tire text anomaly detection method based deep learning and statistics strategy, the mechanism can accurately and robustly detect the region and content of abnormal text. The mechanism is efficient (with a detection time of less than 9 s per lap for one tire) and accurate ( <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$&gt;$</tex-math> </inline-formula> 96%), indicating that it has great potential in the first inspection or quality inspection step of tire manufacturing in industrial scenarios.

Tongue feature dataset construction and real-time detection.

Tongue diagnosis in traditional Chinese medicine (TCM) provides clinically important, objective evidence from direct observation of specific features that assist with diagnosis. However, the current interpretation of tongue features requires a significant amount of manpower and time. TCM physicians may have different interpretations of features displayed by the same tongue. An automated interpretation system that interprets tongue features would expedite the interpretation process and yield more consistent results. This study applied deep learning visualization to tongue diagnosis. After collecting tongue images and corresponding interpretation reports by TCM physicians in a single teaching hospital, various tongue features such as fissures, tooth marks, and different types of coatings were annotated manually with rectangles. These annotated data and images were used to train a deep learning object detection model. Upon completion of training, the position of each tongue feature was dynamically marked. A large high-quality manually annotated tongue feature dataset was constructed and analyzed. A detection model was trained with average precision (AP) 47.67%, 58.94%, 71.25% and 59.78% for fissures, tooth marks, thick and yellow coatings, respectively. At over 40 frames per second on a NVIDIA GeForce GTX 1060, the model was capable of detecting tongue features from any viewpoint in real time. This study constructed a tongue feature dataset and trained a deep learning object detection model to locate tongue features in real time. The model provided interpretability and intuitiveness that are often lacking in general neural network models and implies good feasibility for clinical application.

A heuristic autonomous exploration method based on environmental information gain during quadrotor flight

Autonomous exploration is a widely studied fundamental application in the field of quadrotor, which requires them to automatically explore unknown space to obtain complete information about the environment. The frontier-based method, one of the representative works on autonomous exploration, drives autonomous determination by the definition of frontier information so that complete information about the environment is available to the quadrotor. However, existing frontier-based methods are able to accomplish the task but still suffer from inefficient exploration, and how to improve the efficiency of autonomous exploration is the focus of research nowadays. Slow frontier generation affecting real-time viewpoint determination and insufficient determination methods affecting the quality of viewpoints are typical of these problems. Therefore, to overcome the aforementioned problems, this article proposes a two-level viewpoint determination method for frontier-based autonomous exploration. First, a sampling-based frontier detection method is presented for faster frontier generation, improving the immediacy of environmental representation compared to traditional traversal-based methods. Second, the access to environmental information during flight is considered for the first time, and an innovative heuristic evaluation function is designed to decide on high-quality viewpoint as the next local navigation target in each exploration iteration. Extensive benchmark and real-world tests are conducted to validate our method. The results confirm that our method optimizes the frontier search time by 85%, the exploration time by around 20%–30%, and the exploration path by 25%–35%.

Air Pollution Checker Using Image Processing

In this paper we present a new method to visualize air pollutant through image processing. In order to achieve a realistic effect, we further enhance thus above obtained images in spatial domain. In the proposed method we map the densities of air pollutants to different gray levels, and visualize them by blending those gray levels with background images. The proposed method can visualize large-scale air pollution data from different viewpoints in real time and provide the resulting image with any resolution theoretically, which is very important and favorable for the Internet transmission. Keywords: Machine Learning; Air Pollution; Air Pollution Prediction,images

Real‐time Deep Radiance Reconstruction from Imperfect Caches

AbstractReal‐time global illumination is a highly desirable yet challenging task in computer graphics. Existing works well solving this problem are mostly based on some kind of precomputed data (caches), while the final results depend significantly on the quality of the caches. In this paper, we propose a learning‐based pipeline that can reproduce a wide range of complex light transport phenomena, including high‐frequency glossy interreflection, at any viewpoint in real time (&gt; 90 frames per‐second), using information from imperfect caches stored at the barycentre of every triangle in a 3D scene. These caches are generated at a precomputation stage by a physically‐based offline renderer at a low sampling rate (e.g., 32 samples per‐pixel) and a low image resolution (e.g., 64×16). At runtime, a deep radiance reconstruction method based on a dedicated neural network is then involved to reconstruct a high‐quality radiance map of full global illumination at any viewpoint from these imperfect caches, without introducing noise and aliasing artifacts. To further improve the reconstruction accuracy, a new feature fusion strategy is designed in the network to better exploit useful contents from cheap G‐buffers generated at runtime. The proposed framework ensures high‐quality rendering of images for moderate‐sized scenes with full global illumination effects, at the cost of reasonable precomputation time. We demonstrate the effectiveness and efficiency of the proposed pipeline by comparing it with alternative strategies, including real‐time path tracing and precomputed radiance transfer.

Real-time viewpoint generation-based 3D display with high-resolution, low-crosstalk, and wide view range

A real-time viewpoint generation-based 3D display with low crosstalk and high resolution is proposed. The proposed 3D display consists of a pattern image display (PID), a lenticular lens, and a transparent liquid crystal display (LCD) panel. For each eye of the observer, the PID can provide a pattern image with stripes that have a calculated pitch and displacement according to the eye position. The light from the stripes can be concentrated 3D display eye tracking by the lenticular lens and generate a viewpoint at the eye position. The transparent LCD panel is used to provide parallax images. This transparent LCD panel will not change the direction of light propagation and the location of the generated viewpoints. Then the whole parallax image on the transparent LCD panel can be projected by the PID and the lenticular lens to the viewpoint, so the resolution will not suffer like that in the convention 3D displays. Because the viewpoints are real-time generated according to the position of the observer’s eyes, the observer’s eyes will completely coincide with the position of the generated viewpoint. So crosstalk can be limited at a low level, and the viewing range is almost unlimited.

Revelation of the birth and extinction dynamics of solitons in SWNT-mode-locked fiber lasers

The dispersive Fourier transform (DFT) technique opens a fascinating pathway to explore ultrafast non-repetitive events and has been employed to study the build-up process of mode-locked lasers. However, the shutting process for the mode-locked fiber laser seems to be beyond the scope of researchers, and the starting dynamics under near-zero dispersion remains unclear. Here, the complete evolution dynamics (from birth to extinction) of the conventional soliton (CS), stretched pulse (SP), and dissipative soliton (DS) are investigated by using the DFT technique. CS, SP, and DS fiber lasers mode locked by single-walled carbon nanotubes (SWNTs) are implemented via engineering the intracavity dispersion map. The relaxation oscillation can always be observed before the formation of stable pulse operation due to the inherent advantage of SWNT, but it exhibits distinct evolution dynamics in the starting and shutting processes. The shutting processes are dependent on the dispersion condition and turn-off time, which is against common sense. Some critical phenomena are also observed, including transient complex spectrum broadening and frequency-shift interaction of SPs and picosecond pulses. These results will further deepen understanding of the mode-locked fiber laser from a real-time point of view and are helpful for laser design and applications.

A multi-scale VR navigation method for VR globes

ABSTRACTThe combination of virtual reality (VR) and virtual globes – VR globes – enables users to not only view virtual scenes in an immersive manner at any location on Earth but also directly interact with multi-scale spatial data using natural behaviors. It is an important direction for the future development of 3D GIS and geovisualization. However, current VR navigation are primarily based on small real spaces. For virtual globes, which are 3D multi-scale globe environment, the realization of VR navigation in the multi-scale virtual globe space within a limited real space is the first problem that needs to be addressed. A multi-scale VR navigation method that consists of two algorithms is proposed in this study. The first algorithm maps the real space to the virtual globe space and connects the VR user with the VR viewpoint. The second algorithm is an octree structure-based viewpoint correction algorithm that is proposed to correct the location of the moving VR viewpoint in real time. The proposed method is validated by experimentation. The experimental results indicate that the proposed method enables a VR user to interactively view the 3D multi-scale globe environment and lays a foundation for human–computer interaction in VR globes.

Evaluation of viewpoint translation of telepresence robot by trochoid trajectory

We proposed a simple link mechanism running on geometrically complete trochoidal trajectory as a novel holonomic and omnidirectional vehicle. In telepresence the accuracy on the position of the viewpoint in real time determines the reality. In this report, using the telepresence robot that runs using the Omni wheel, we evaluate head followability when traveling on the trochoid trajectory.

ClickSmart: A Context-Aware Viewpoint Recommendation System for Mobile Photography

In this paper, we propose ClickSmart , a viewpoint recommendation system that can assist a user in capturing high-quality photographs at well-known tourist locations. ClickSmart can provide real-time viewpoint recommendation based on the preview on the user’s camera, current time, and user’s geolocation. It makes use of publicly available geotagged images along with the associated metadata for learning a recommendation model. We define view-cells , macroblocks in geospace, and propose the concepts of popularity , quality , and uniqueness of view-cells from the viewpoint perspective. Viewpoint recommendation is generated at the granularity of a view-cell and is based on its popularity , quality , and uniqueness , which are estimated using social media cues associated with images. We further observe that contextual information such as time and weather conditions play an important role in photography, and therefore augment the recommendation system with the associated context. ClickSmart also takes into account the presence of people in the view for making the recommendation. It can provide two kinds of recommendations, quality based and uniqueness based. Although both were found effective in the experimental evaluation, our user study showed that uniqueness -based recommendation was preferred more by skilled photographers compared with amateurs.

A Middleware Based Architecture for the Industrial Internet of Things

Due to its potential, the Internet of Things (IoT) begins to be used in the industrial field. Nevertheless, although the concept called the Industrial Internet of Things does not know a widely accepted definition, it is based on the existing technology, and introduces the IoT in the industrial environment through the compliance with specific additional requirements and by integrating various fieldbuses with distinctive features, from the real-time point of view. The aim of this paper is to propose a practical architecture for the Industrial Internet of Things. The proposed architecture is designed around the Data Distribution Service for Real Time System (DDS) middleware protocol, and it is organized on four levels. Between these levels, it is defined standard interface, and they can be developed independently. For this reason, the architecture is scalable because at the middleware level, the DDS can be replaced by other middleware systems. At the final, it is performed a comparison between several middleware systems used in the proposed architecture.

Architectural Design Virtual Simulation Based On Virtual Reality Technology

As the modeling of the architectural system is difficult and inaccurate, the 3D modeling of the light in the architectural system is studied by using Java3D technology. Using java3D perspective projection technique, the observation coordinate system is determined, and real-time viewpoint setting as well as roaming processing is realized. Through multiple view technique, we can observe the movement of building bodies from multiple aspects to obtain vivid simulation effects of light' behavior in the architectural system. Simulation results show that the virtual simulation platform developed based on java3D can improve the accuracy and naturalness of architectural system simulation.

Large-Scale Virtual Simulation of the Solar System Based on Java3D

As the modeling of the solar system is difficult and inaccurate, the 3D modeling of the planets in the solar system is studied by using Java3D technology. To make the revolution of simulated planets meet Kepler's laws while traveling around the sun, an improved approximate algorithm to calculate the optimum angle and the vertex coordinates of each planet is proposed. Using java3D perspective projection technique, the observation coordinate system is determined, and real-time viewpoint setting as well as roaming processing is realized. Through multiple view technique, we can observe the movement of celestial bodies from multiple aspects to obtain vivid simulation effects of planets' behavior in the solar system. Simulation results show that the virtual simulation platform developed based on java3D can improve the accuracy and naturalness of solar system simulation.

Immersive 3D Telepresence

Cutting-edge work on 3D telepresence at a multinational research center provides insight into the technology's potential, as well as into its remaining challenges. The first Web extra at http://youtu.be/r4SqJdXkOjQ is a video describing FreeCam, a system capable of generating live free-viewpoint video by simulating the output of a virtual camera moving through a dynamic scene. The second Web extra at http://youtu.be/Dw1glKUKs9A is a video showing a system designed to capture the enhanced 3D structure of a room-sized dynamic scene with commodity depth cameras, such as Microsoft Kinects. The third Web extra at http://youtu.be/G_VPzXRrmIw is a video demonstrating a system that adapts to a wide variety of telepresence scenarios. By combining Kinect-based 3D scanning with optical see-through HMDs, the user can precisely control which parts of the scene are real and which are virtual or remote objects. The fourth Web extra at http://youtu.be/n45N5AHsoCI is a video demonstrating a method based on moving least squares surfaces that robustly and efficiently reconstructs dynamic scenes captured by a set of hybrid color+depth cameras. The reconstruction provides spatiotemporal consistency and seamlessly fuses color and geometric information. The video also illustrates the formulation on a variety of real sequences and demonstrates that it favorably compares to state-of-the-art methods. The fifth Web extra at http://youtu.be/OSl3f2qZzKs is a video demonstrating a 3D acquisition system capable of simultaneously capturing an entire room-sized volume with an array of commodity depth cameras and rendering it from a novel viewpoint in real time. The sixth Web extra at http://youtu.be/zKWByH7evo0 is a video demonstrating a gaze-correction approach based on a single Kinect sensor that preserves both the integrity and expressiveness of the face as well as the fidelity of the scene as a whole, producing nearly artifact-free imagery. The method is suitable for mainstream home video conferencing: it uses inexpensive consumer hardware, achieves real-time performance, and requires just a simple and short setup.

Camera movement for chasing a subject with unknown behavior based on real-time viewpoint goodness evaluation

We propose a method that automatically generates a smooth chase camera movement to follow a subject, a user-controlled character or a character with unknown behaviors, in a 3D environment freely in real time. We consider three objectives in generating the smooth-camera movement: to avoid collisions with obstacles, to avoid subject occlusions, and to choose a good viewpoint for looking at the subject. We evaluate the goodness of viewpoints by using a viewpoint entropy map and choose the best viewpoint as the goal position of the camera in real time. Afterwards, we move the camera toward the goal position by following the shortest path, found by the A* algorithm, on a roadmap graph. The resulting camera movement has a high degree of freedom and fulfills the three objectives above. Our method is effective for third-person-view 3D applications in tracking the real-time movement of user-controlled characters in exploring a 3D environment.

Real‐time interactive fusion of physical/electronic auscultation simulation and 3D CT/MRI‐based anatomical modeling applied to training in lung sounds detection, discrimination and classification

Objective: To test the instructional effectiveness of simultaneous coupling of the spatial detection, discrimination and classification of simulated lung sounds to real‐time viewpoint tracking of displayed digital 3D anatomical models over the current method of auscultation alone. Methods: An integration of a computer‐based clinical training system comprising two complementary functions: 3D anatomical visualizations, and physiological diagnostic sounds simulation using a physical manikin. These functions are linked in real‐time by the student stethoscope's 3D position/orientation in space on the surface of the manikin. Thus, during any given simulation drill using as input a pre‐defined diagnostic lung sound, a student will sample signs from the manikin and simultaneously view the case's prescribed condition or pathology volumetrically reconstructed and visualized from de‐identified patient MRI/CT image sets carefully selected to represent the given abnormality. Expected results: Active student tracking, perception of diagnostic sounds and visualizations of the corresponding anatomical structures will significantly increase student learning over methods using auscultation without visualization.This research is supported by the Department of Anatomy and the Dean's Office, Des Moines University.

Immersive Video Teleconferencing with User-Steerable Views

Existing video teleconferencing techniques suffer from limited field of view, low resolution, and fixed viewpoint. We present a set of novel techniques to overcome these limitations. Based on the light field rendering concept, we utilize an array of cameras to capture the participant(s) from multiple view angles. Based on either assumed or estimated scene geometry, these images are assembled to create a high-resolution seamless image from a user-controllable viewpoint. Two different camera configurations, one dense and one sparse, are presented; the dense format is optimized for high-fidelity view synthesis while the sparse configuration is for expanded viewing volume. For the dense configuration, we provide a detailed analysis on the number of camera images required. For the sparse configuration, we present a robust technique to estimate an approximation of the scene geometry to provide smooth transition when the viewpoint is changed.The novelty of our approach is that it allows the users to electronically steer the viewpoint in real time for a live scene. Therefore it can be used in 3D teleconferencing systems to generate stereoscopic views, or in group teleconferencing to provide virtual camera views that minimize the overall perspective distortions. We demonstrate the effectiveness of our approach with a point-to-point teleconferencing system distributed in several locations across the continental United States.

화자 인증 기능이 포함된 실시간 원격 도어락 제어 시스템 개발에 관한 연구

The paper attempts to design and implement the system which can remotely check visitors` speech or Image by a mobile phone. This system is designed to recognize who a visitor is through the automatic calling service, not through a short message, via the mobile phone, even when the home owner is outside. In general, door locks are controlled through the home Server, but it is more effective to control door locks by using DTMF signal from a real-time point of view. The technology suggested in this paper makes it possible to communicate between the visiter and the home owner by making a phone call to tile home owner`s mobile phone automatically when the visiter visits the house even if the home owner is outside, and if necessary, it allows for the home owner to control the door lock remotely. Thanks to the system, the home owner is not restricted by time or space for checking the visitor`s identification and controlling the door lock. In addition, the security system is improved by changing from the existing password form to the combination of password and speaker verification lot the verification procedure required for controlling the door lock and setting the environment under consideration of any disadvantages which may occur when the mobile Phone is lost. Also, any existing problems such as reconnection to tile network for controlling tile door lock are solved by controlling the door lock in real time by use of DTMF signal while on the phone.

Incomplete 3-D multiview representation of video objects

This paper introduces a new form of representation for three-dimensional (3-D) video objects. We have developed a technique to extract disparity and texture data from video objects that are captured simultaneously with multiple-camera configurations. For this purpose, we derive an "area of interest" (AOI) for each of the camera views, which represents an area on the video object's surface that is best visible from this specific camera viewpoint. By combining all AOIs, we obtain the video object plane as an unwrapped surface of a 3-D object, containing all texture data visible from any of the cameras. This texture surface can be encoded like any 2-D video object plane, while the 3-D information is contained in the associated disparity map. It is then possible to reconstruct different viewpoints from the texture surface by simple disparity-based projection. The merits of the technique are efficient multiview encoding of single video objects and support for viewpoint adaptation functionality, which is desirable in mixing natural and synthetic images. We have performed experiments with the MPEG-4 video verification model, where the disparity map is encoded by use of the tools provided for grayscale alpha data encoding. Due to its simplicity, the technique is suitable for applications that require real-time viewpoint adaptation toward video objects.